Video overlay systems

A system may include computing equipment and a handheld video overlay accessory. A video overlay accessory may include one or more light sources, an image sensor, and processing circuitry. The video overlay accessory may include an image projector that projects images onto external objects. The video overlay accessory may include a partially transparent beam splitter and a display that projects display content onto the partially transparent beam splitter. A user that views a scene through the partially transparent beam splitter may view the display content apparently overlaid onto the scene. Light sources in the video overlay accessory may be laser diodes or other light-emitting diodes that project tracking spots such as infrared tracking spots on to external objects. The image sensor may be used to capture images of the tracking spots. The processing circuitry may determine distance and position information based on the captured images.

BACKGROUND

This relates generally to systems that display visual information and, more particularly, to systems that overlay visual content on real-world objects.

Electronic devices often have input-output components. For example, an electronic device may contain an output component such as a display providing visual output to a user. In conventional systems, devices commonly have a display that presents device-generated visual content to the user. In some systems, the device-generated visual content may be altered based on the information about the environment around the device.

For example, a map may be displayed on the display that is oriented based on the orientation of the device in the real world. As another example, images of a real-world scene are sometimes captured and displayed on the device display while device-generated visual content is overlaid on the captured images that are displayed. However, it can be challenging to present device-generated display content combined with real-world scene content using conventional image capture and display systems.

It may therefore be desirable to provide improved image overlay systems.

DETAILED DESCRIPTION

An illustrative image overlay system is shown inFIG. 1. As shown inFIG. 1, system10may include an accessory14that includes input devices-output devices18, processing circuitry20, and positioning devices22. Accessory14may be an electronic device such as a handheld controller or handheld display device. A handheld display device may be a video overlay device (video overlay accessory) such as a video overlay projection device (i.e., a device that projects images onto real-world objects) or a video overlay device having a beam splitter that presents images that appear to be located in a real-world scene.

Input-output devices18may include input components (e.g., buttons, switches, cameras or other input components) and output components such as a display or an image projector. Positioning devices22may include light-emitting devices such as light source34that generate light-based marks on real-world objects that can be tracked by accessory14(e.g., using an image sensor or camera associated with input-output devices18) or mechanical or electronic positioning devices such as accelerometer21or global-positioning-system receivers that help track the position, orientation, and movement of accessory14.

Accessory14may optionally be connected to external electronic equipment12such as a computer or game console. Accessory14may, for example, be coupled to equipment12using communications path16. Path16may be a wireless path or a wired path (e.g., a Universal Serial Bus path). Input such as user input, accessory position and orientation data and/or scene object position data from accessory14may be used to control equipment12. For example, computing equipment12may supply display data to accessory14for display to a user that is based on input from accessory14(e.g., based on the position, orientation or motion of the accessory and/or based on the position or orientation objects in the vicinity of accessory14).

The operation of device14may be controlled using input that is gathered from input-output devices18and/or positioning devices22. For example, input from devices18and/or22may be processed by processing circuitry20. Processing circuitry20may then direct input-output devices18to take suitable actions in response to the user input. For example, processing circuitry20may use input-output devices18to provide visual information to a user that depends on the input data received by components18and/or22. Processing circuitry20may relay user input, accessory position and orientation data and/or scene object position data to external equipment12via path16, if desired.

An illustrative configuration that may be used for accessory14is shown inFIG. 2. As shown inFIG. 2, accessory14may be implemented as a video overlay projection device. In the example ofFIG. 2, accessory14includes light source34for projecting optical tracking spots, image projector30for projecting images such as video images onto real-world objects, light sensor32, and user input component38mounted in housing structure36.

Image projector30may include light-generating elements that project images such as projected image40onto real-world objects such as a wall or other object. User input component38(e.g., a button or trigger) can be operated by a user to operate accessory14(e.g., to initiate or terminate video overlay projection operations). Light source34(e.g., a near-infrared light source, optical light source or other light source) may include a laser diode or other light-emitting diode and, if desired, optical components (e.g. a grating or a slit) that project optical tracking spots42on real-world objects. Light source34may be used to generate one, two, three, four or more than four tracking spots42.

Light sensor32may be a camera having an image sensor (e.g., an image sensor integrated circuit having an array of image pixels such as complementary metal-oxide-semiconductor (CMOS) image pixels, charge-coupled-device (CCD) image pixels, or image pixels based on other types of image sensor technology) that is used to capture images of the real-world scene. Images of tracking spots42in the captured images may be used to determine the distance between accessory14and the object onto which image40is being projected. Processing circuitry such as circuitry20of accessory14may be used to determine the distance between one or more spots42in the captured images (e.g., by determining the number of image pixels between each spot42in a captured image). Circuitry20may use the determined distances between spots42and known angles such as angles A and B between the projected beams that produce spots42to determine the physical distance from accessory14to a given object.

Accessory14may include multiple light sources34(e.g., multiple laser diodes) for producing multiple associated spots42or may include a single light source34having a light-generating element37(seeFIG. 4below for a cross-sectional view of light source34) and an optical element35(e.g., a grating or slit) that generates two or more light beams41that produce spots42. Light-generating element37may be, for example, a laser diode or other light-emitting diode. The pitch of the grating or the width of the slit may be configured to generate a desired angle (e.g., angles A and B) between beams41. If desired, optical element35may be a grating structure that includes a two-dimensional grating pattern that produces a grid of beams41that spread at a common spot-generating angle to generate a grid of spots42covering a given field-of-view.

Camera32may include an image pixel array having a pixel projection angle that is different from the spot-generating angle. In this way, the number of pixels between spots42in captured images will change when the distance between accessory14and objects in the field-of-view changes and the determined distance between spots42in captured images may be used to determine physical distances to objects.

Processing circuitry20may use the determined distance between accessory14and the given object to focus image40and or to determine which display content is to be displayed in image40.

If desired, light source34may be used to generate four or more beams41and corresponding spots42. In this type of configuration, images of the four or more spots42may be used to determine an angular position of accessory14relative to an object. For example, when oriented perpendicular to an object, four or more evenly spaced beams will produce four or more evenly spaced spots in an image. At any angular position that is not perpendicular, the distance between the spots will be unevenly spaced. Based on measurements of this uneven distribution of spots in a captured image, the angle between accessory14and an object may be determined. This type of angular orientation data may be used by circuitry20to generate image corrections using projector30that can compensate for a detected tilt with respect to the object.

Processing circuitry20may also be used to generate a depth map of a real-world scene using the evenly spaced grid of spots42. Objects that are relatively nearer to accessory14will have relatively closely spaced spots42in comparison with relatively further spaced spots42on a relatively further away object. Processing circuitry20may use the distribution of spots in a captured image to determine the relative (or absolute) distances to objects in the field-of-view.

Accessory14may use positioning devices22and camera32to track the motion of accessory14. For example, as shown inFIG. 3, images of spots42may be used to track the motion of accessory14from a first position such as position52to a second position such as position52(in direction50) as spots42travel across real-world objects in a scene. Tracking the motion of accessory14using spots42(in addition to, or instead of tracking the motion of accessory14using accelerometer21(FIG. 1)) may allow relatively faster tracking of the motion of accessory14that prevents video content displayed in image40from lagging behind the motion of accessory14.

Accessory14may generate display content to be displayed in image40based on the position of accessory14. In the example ofFIG. 3, no display content is included in image40at position52. A display object such as display object56may be added to image40when accessory14is in position54. Accessory14may be used to display object56at a location58that is independent of the position of accessory14. For example, as a user moves accessory14around position54, object56may remain at position58and may therefore move out of and into image40based on the motion of accessory14.

In one example, object56may be a structural member of a building that is not visible to a person in the building (e.g., a wall stud inside of a wall). Accessory14(or computing equipment12) may include stored building plans that include the position of the wall stud in the wall. When accessory14is scanned over that position in the wall, accessory14may project an image of the wall stud (based on the stored building plans) at that location.

In another example, object56may be an internal anatomical member of a person or an animal such as a muscle, a tendon, a bone, etc. Accessory14may include stored anatomical data such as textbook anatomical drawings or person-specific x-ray, MRI, or other imaging data that include the position of the anatomical member in the person or animal. When accessory14is scanned over that position in the person or animal, accessory14may project an image of the member onto the person or animal at that position.

In another example, object56may be structure in a cave. Accessory14may include stored cave-structure data such as image data, radar data, or other data that include the position of the cave structures in the cave. Accessory14may be used to provide a virtual tour of the cave by projecting images of the cave structures in a dark room when accessory14is aimed at the position of those objects in the virtual cave.

The examples ofFIGS. 2 and 3in which accessory14includes an image projector are merely illustrative. If desired, accessory14may be implemented as video overlay device having a beam splitter that presents images that appear to be located in a real-world scene as shown inFIG. 4.

In the example ofFIG. 4, accessory14includes display60and beam splitter62along with light source34and camera32. Display60may be a liquid-crystal display (LCD) that generates images to be presented to a user such as user68. Images generated by display60may be projected onto beam splitter62. Beam splitter62may be a partially transparent beam splitter that allows user68to see through beam splitter62and view real-world objects such as object62through beam splitter62. By allowing user68to see objects64through beam splitter62and projecting an image onto beam splitter62, images generated by display60may appear to user68to be overlaid onto objects such as object64. Image content in images generated using display60may be generated in response to images of spots42on object64that have been captured using camera32. For example, processing circuitry20may overlay an image of a display object onto object64when accessory14is scanned to a particular position or when accessory14is moved to a particular distance from an object such as object64.

As shown inFIG. 4, accessory14may include an extended portion such as plastic handle shaped portion66that includes a user input component such as trigger38. Trigger38may be operated by user68when a particular object appears in an image generated by display60.

For example, a user may view a portion such as portion80of a real world scene through beam splitter62as shown inFIG. 5. When the user moves beam splitter62in direction72, a portion80′ of the real-world scene containing some or all of an object64(e.g., a chair) may become visible through beam splitter62. In some positions, a real-world object64may be visible through beam splitter62while no visual display objects are presented. When the user moves the beam splitter62in direction74, a portion80″ of the real-world scene containing some or all of one or more objects64(e.g., a table) may be visible through beam splitter62and display60may overlay a display object70onto the real-world scene. Display object70may be located at a position with respect to the real-world scene that does not change when the position of accessory14is changed. In this way, the user may be allowed to scan the real-world scene while trying to find display object70. Accessory14may instruct the user to take a particular action when a particular display object is located. For example, user68may actuate trigger38when a display object70such as a video-game enemy or an educational object is recognized.

Processing circuitry20may be used to identify edges of objects64in a real-world scene in additional to generating a depth map of the scene. Using the edge information and depth information generated using images captured using camera32, overlaid display objects70may be scaled video objects that blend into the viewed real-world scene as if there were an integral portion of the environment. For example, in a video gaming operational mode for accessory14, a display object70may be a video game character that appears to emerge from behind an object64or from within an object64.

An accessory such as accessory14that includes light source34, camera32, processing circuitry20, display60and beam splitter62may provide a user with similar capabilities to those described in the examples above in connection withFIGS. 2 and 3. In particular, display object70may be a structural member of a building that is displayed using beam splitter62when accessory14is scanned over a particular position in the building (e.g., based on the stored building plans), object70may be an internal anatomical member of a person or an animal that is displayed using beam splitter62when accessory14is scanned over a particular position in the person or animal (based on text book or medically obtained images of the person or animal), object70may be structure in a cave that is displayed using beam splitter62when accessory14is scanned over a particular position in a room when accessory14is aimed at the position of that structure in a virtual cave, or object70may be a video game object or educational software object to be located by the user (as examples).

Illustrative steps that may be used in operating a system such as system10having an accessory such as accessory14ofFIG. 1are shown inFIG. 6.

At step100, display content may be provided to a user. Providing the display content to the user may include projecting display content onto external objects using a projector such as projector30ofFIG. 2or may include overlaying display content in front of real-world objects using a display such display60and a beam splitter such as beam splitter62ofFIG. 4.

At step102, device position and orientation data and scene object position data may be gathered using the accessory.

At step104, altered display content may be provided to the user based on the gathered device position and orientation data and/or the gathered scene object position data. Altering the display content may include focusing the display content based on the position and orientation of the accessory and/or scene objects, updating display content based on the motion and/or the position of the accessory, or otherwise altering the display content based on the position data.

Illustrative steps that may be used in gathering device position and orientation data and scene object position data as described above in connection with step102ofFIG. 6are shown inFIG. 7.

At step110, positioning device data such as accelerometer data may be gathered.

At step112, one or more narrow-beam light sources such light source34ofFIGS. 2and/or4may be activated in order to project tracking spots such as spots42ofFIGS. 2,3,4, and5onto one or more external objects.

At step114, image data that includes projected tracking spot information may be gathered using, for example, camera32ofFIG. 2and/orFIG. 4. Projected tracking spot information may include pixel positions of tracking spots in the gathered image data.

At step116, absolute and/or relative positions of the projected tracking spots may be determined using the gathered image data. Determining the absolute and/or relative position of the projected tracking spots may include determining a number of pixels between spots42in captured images and determining the absolute and/or relative position of the projected tracking spots using the determined number of pixels and known projection angles between narrow-beams that produced the tracking spots.

At step118, device position and orientation data and scene object position data may be generated using the gathered positioning device data and/or the determined positions of the projected tracking spots. Generating device position and orientation data and scene object position data using the gathered positioning device data and/or the determined positions of the projected tracking spots may include generating device motion data that represents determined changes over time of the device position and/or orientation. Generating device position and orientation data and scene object position data using the gathered positioning device data and/or the determined positions of the projected tracking spots may include generating a depth map of the real-world scene, determining the location of edges of objects in the scene or determining an angle between the accessory and an object in the scene (as examples).

FIG. 8shows, in simplified form, a typical processor system300, such as accessory14ofFIG. 1. Processor system300is exemplary of a system having digital circuits that could include imaging device200(e.g., an image sensor in camera30ofFIG. 2and/orFIG. 4). Without being limiting, such a system could include a computer system, still or video camera system, scanner, machine vision, vehicle navigation, video phone, surveillance system, auto focus system, star tracker system, motion detection system, image stabilization system, video gaming system, video overlay system, and other systems employing an imaging device.

Processor system300, which may be a digital still or video camera system, may include a lens such as lens396for focusing an image onto a pixel array such as pixel array201when shutter release button397is pressed. Processor system300may include a central processing unit such as central processing unit (CPU)395. CPU395may be a microprocessor that controls camera functions and one or more image flow functions and communicates with one or more input/output (I/O) devices391over a bus such as bus393. Imaging device200may also communicate with CPU395over bus393. System300may include random access memory (RAM)392and removable memory394. Removable memory394may include flash memory that communicates with CPU395over bus393. Imaging device200may be combined with CPU395, with or without memory storage, on a single integrated circuit or on a different chip. Although bus393is illustrated as a single bus, it may be one or more buses or bridges or other communication paths used to interconnect the system components.

Various embodiments have been described illustrating video overlay systems having a portable accessory such as accessory14of (for example)FIG. 1with a combination of a light source34, camera32, and video overlay components such as display60and beam splitter62ofFIG. 4or image projector30ofFIG. 2.

Light sources may include one or more laser diodes or one or more other light-emitting diodes that project beams of light at known angles onto objects in a scene. A camera may include an array of image pixels for capturing images of spots that have been generated by the projected beams. The processing circuitry may be used to determine distances to objects and locations of edges of objects using the images of the projected spots.

An image projector may be used to project display images directly onto objects in the scene or a display may generate display images to be viewed on a beam-splitting structure that allows the user to view the display images on the beam splitter in addition to viewing portions of the real world scene through the beam splitter. In this way display objects may be scaled and overlaid onto a portion of a real world scene.

This type of system may allow a user to interact with their surroundings and to project or view images on surfaces or in other locations in the environment. A system of this type may be used to generate images of the internal construction of objects in a real-world scene, or to overlay other images that have been scaled to the size and position of objects in the real-world scene. In this way, a user may be provided with the ability to locate otherwise invisible portions of an object and to align images of the invisible portions with visible portions of the object.

The foregoing is merely illustrative of the principles of this invention which can be practiced in other embodiments.