Generating still images and video by capture of images projected by light passing through a display screen

Methods and systems for generating still images and video by capture of images projected by light passing through a display screen are disclosed. According to an aspect, a method includes controlling light emitting components of a display screen to activate in a sequence for displaying a sequence of images. For example, in an organic light emitting diode (OLED) display screen, the method may include controlling a subset of the display screen OLEDs to activate in sequence to display a sequence of images. The displayed sequence of images may appear to a viewer of the display screen to be a still image or a video. The method may control light emitting components different subsets of the LEDs to be activated in the sequence, such that different areas of the display screen display images in each part of the sequence.

BACKGROUND

1. Field of the Invention

The present invention relates to displays and image capture equipment, and more specifically, to display screens for displaying images and image capture devices for use in generating still images and video by capture of images projected by light passing through the display screens.

2. Description of Related Art

Display screens are often integrated into electronic equipment for displaying text, still images, and video to a user of the electronic equipment. Efforts on ongoing to support and increase the functionality of display screens, especially in mobile computing applications, such as smartphone technologies, where it is desirable to reduce the size and weight of such devices. By integrating the functionality of other components into a display screen, the size and weight of electronic equipment including the display screen may be reduced.

Image capture functionality is a feature of electronic devices that has become commonplace for capturing still images and video. For example, image capture devices have been integrated into many electronic devices, such as, smart phones, automated teller machines (ATMs), laptop computers, and desktop computers. In such devices, an image capture device and a display screen are often positioned on a side of a housing of the devices such that the display screen can be viewed by a user, while at the same time, an image of the user and any background scenery can be captured by the image capture device. Such simultaneous use is common in video conferencing applications. Display screens and image capture devices can require a considerable area of a side of the electronic device. Accordingly, for at least this reason, it is desirable to integrate display screens and image capture devices in electronic devices for reducing space and size consumption.

BRIEF SUMMARY

One or more embodiments of the present invention provide methods and systems for generating still images and video by capture of images projected by light passing through a display screen. In accordance with embodiments of the present invention, a method includes controlling light emitting components of a display screen to activate in a sequence for displaying a sequence of images. For example, the method may include controlling a subset of the display screen organic light emitting diodes (OLEDs) to activate in sequence to display a sequence of images. The displayed sequence of images may appear to a viewer or user of the display screen to be a still image or a video. The method may control different subsets of the light emitting components to be activated in the sequence, such that different areas of the display screen display images in each part of the sequence.

The method may also include controlling other light emitting components of the display screen that are not activated for displaying the images to be inactive such that a portion of the display screen is at least partially transparent for allowing light to pass therethrough. The light passing through the display screen may project a sequence of images on an image capture device. Continuing the aforementioned example, the OLEDs may be transparent OLEDs that are at least partially transparent when inactive. When the OLEDs are controlled to be inactive, the corresponding portions of the display screen become at least partially transparent for allowing light to pass therethrough towards an image capture device positioned behind the display screen. The light may project a sequence of images on the image capture device, and the image capture device may capture the sequence of images at the same or similar rate as the display screen displays images. The images may be of a scene including, for example, a viewer of the display screen and any scenery near the viewer. In an example, different groups of the light emitting components may be controlled to become inactive in a sequence for allowing different portions of an image of the scene to be projected onto the image capture device. The method may include controlling the image capture device to capture the different portions of the image of the scene. The captured image portions may be stitched together for generating a composite image of the scene.

DETAILED DESCRIPTION

Exemplary systems and methods for generating still images and video by capture of images projected by light passing through a display screen in accordance with embodiments of the present invention are described herein. Particularly, described herein are systems and methods for controlling light emitting components of a display screen to activate in a sequence for displaying a sequence of images or video. An example system may be an electronic device including a display screen and an image capture device positioned behind the display screen within the electronic device. An example method may include controlling other light emitting components of the display screen that are not activated for displaying the images to be inactive such that a portion of the display screen is at least partially transparent for allowing light to pass therethrough to thereby project a sequence of images on the image capture device. The image capture device may capture the sequence of projected images. The example method may also include generating a still image based on the captured sequence of projected images. For example, the captured sequence of projected images may represent components of a scene, and the component may be stitched together for generating a still image of the scene.

As referred to herein, the term “display screen” is a component of a display device configured for displaying images to a viewer. An example of a display screen includes an organic light emitting diode (OLED) display, which may include an emissive electroluminescent layer of organic compounds which emit light in response to an electric current. In an OLED display, the layer of organic semiconductor material may be situated between two electrodes, one or both of which may be transparent. Other examples of display screens include, but are not limited to, a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT-LCD), a flexible display, a three-dimensional display, a touch screen display, or the like. These display screens can be configured to have portions that are transparent or at least partially transparent in accordance with embodiments of the present invention such that light may pass therethrough.

As referred to here, the term “image capture device” is any suitable device for capturing images of a scene. For example, an image capture device may be a video or digital camera having an electronic image sensor, such as a charge coupled device (CCD) or a CMOS sensor, configured to capture images which can be transferred or stored in a memory or any other suitable storage for later playback or processing. The camera may capture many images in a sequence for storage as video. An image capture device may include a lens for capturing the light from a subject and bringing the light into focus on the electronic image sensor.

As referred to herein, the term “image data” is data suitable for interpretation by a computing device for representing an image. For example, image data may be stored in a digital image file format such as, but not limited to, PNG, JPEG, GIF, Exif, TIFF, RAW, BMP, PPM, PGM, PBM, PNM, and WEBP. A sequence of image, or video, may be stored in a suitable format such as MPEG. Image files may be composed of either pixel or vector (geometric) data that may be rasterized to pixels when displayed in a vector graphic display, such as a suitable display screen. The pixels that constitute an image are ordered as a grid (column and rows), each pixel consisting of numbers representing magnitudes of brightness and color.

As referred to herein, the term “memory” is one or more components configured to store data. For example, memory may store a program for the processing and control of a controller. Memory may also temporarily store input/output data (e.g., still images, video, and the like). Also, the memory may be implemented using any type of suitable storage medium including, for example, but not limited to, flash memory, a hard disk, a multimedia card micro type, a memory card (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, and the like. Further, for example, the memory may be implemented via web storage which performs the storage function of the memory on the Internet, or via any other suitable type of network memory storage.

FIG. 1illustrates a schematic side elevational view of a system100for generating still images and video by capture of images projected by light passing through a display screen102in accordance with embodiments of the present invention. Referring toFIG. 1, the system100may operate in an image capture and display mode in which the display screen102displays a sequence of images or video for view, and in which an image capture device104captures a sequence of images projected thereon by light passing through one or more transparent or at least partially transparent portions of the display screen102. The display screen102may display the sequence of images at a frame rate, or frame frequency, that is suitable for a viewer of the display without showing undesirable image noise or disturbance. For example, the frame rate may be at least 25 frames per second. The image capture device104may also capture images in a sequence at the same rate as the frame rate of the display screen102, or any other suitable image capture rate. The system may generate one or more still images by use of the captured sequence of projected images.

Some or all of the components of the system100may be contained within an enclosure106such as a housing of an electronic device such as, for example, but not limited to, a laptop computer, desktop display, television, mobile computing device, mobile telephone, or other electronic devices. More generally, the system100may be implemented in any electronic device having display and image capture functions. The enclosure106may define an opening or window108on a front side. The display screen102may be positioned along the front side of the enclosure106, adjacent to the opening or window108. The display screen102has a front display side110and a back side112. In this example, the display screen102may be an OLED screen, although any other suitable type of display screen may be used.

The system100may include an image capture controller114configured to control operation in the image capture and display mode in accordance with embodiments of the present invention. For example, the controller114may include a display interface116for interfacing with the display screen102. The controller114may include an image capture device interface118for interfacing with the image capture device104. The interfaces116and118may be drivers or other components suitably configured to control the display screen102and the image capture device104, respectively. The controller114may include suitable hardware, software, firmware, or combinations thereof for implementing processes in accordance with embodiments of the present invention described herein. One or more power sources120may provide the power necessary to operate the display screen102, the image capture device104, and the controller114.

In the image capture and display mode, the system100may display still images or video while capturing still images or video of a scene. For example,FIG. 2illustrates a flowchart of a method for controlling a display screen and image capture device to display images and to generate a still image or frame of a video in accordance with embodiments of the present invention. The method ofFIG. 2is described as being implemented by the system100shown inFIG. 1, although the method may be implemented by any system including a suitable display screen and image capture device.

Referring toFIG. 2, the method ofFIG. 2includes entering200the image capture and display mode. For example, a user of the system100may interact with a user interface122to enter user input for controlling the system100to enter the image capture and display mode. The user interface122may be any suitable component configured to receive and to process user input. For example, the user interface122may be a touch screen display, keyboard, keypad, mouse, track pad, and the like. Alternatively, for example, entry into this mode may be automatically affected by an application residing on the system100and implementing display and image capture functions.

The method ofFIG. 2includes controlling202light emitting components of a display screen to activate in a sequence for displaying a sequence of images. For example, the controller114may be configured to control light emitting components of the display screen102to display a sequence of images or video. The images or video may be stored in a suitable format in a memory124of the system100. As an example, the light emitting components may be LEDs that can be individually activated for emitting different colors of light at varying brightness to appear as an image to a viewer facing the front side110of the display screen102. In accordance with one or more embodiments of the present invention, the display screen102may include a matrix of OLEDs that are each electronically switchable between an “on” state and an “off” state. The display interface118can control application of electrical current to a predetermined set of the OLEDs for switching the OLEDs between the on and off states. The predetermined set of OLEDs may be a subset of the OLEDs in the matrix. In an example, the same subset of OLEDs in the matrix may be activated in a sequence at a predetermined frame rate for displaying a still image or video over a period of time. In another example, different subsets of the OLEDs in the matrix may be activated in a sequence for displaying a sequence of images or video over a period of time. By activating different subsets of OLEDs in the matrix, different areas of the display screen are activated in the sequence for displaying the sequence of images. The subset of OLEDs activated in the sequence may be a majority of the OLEDs in the matrix.

The method ofFIG. 2includes controlling204other light emitting components of the display screen that are not activated for displaying the images to be inactive such that a portion of the display screen is at least partially transparent for allowing light to pass therethrough for projecting a sequence of images on an image capture device. Continuing the aforementioned example of a matrix of OLEDs, the controller114may control a subset of the OLEDs in the matrix that are not activated for displaying the images at202to be inactive such that a portion of the OLED display screen is at least partially transparent for allowing light to pass therethrough for projecting a sequence of images on the image capture device104. The projected sequence of images may each be a portion of a scene near the display screen102. The scene may include a viewer of the display screen102, a user of the system100, and/or any nearly scenery or objects. The different images may be projected onto the image capture device104over a period of time. Images displayed by the display screen102may also be projected onto the image capture device104.

Continuing the example ofFIG. 1, when the system100is operating in the image capture and display mode, light may pass through a transparent portion, or at least partially transparent portion, of the display screen102, and may project an image onto an electronic image sensor of the image capture device104. For example, the display interface118of the controller116can control a subset of OLEDs generally located at a portion124of the display screen102to be in the off state such that this portion of the display screen102is transparent, or at least partially transparent, for allowing light (indicated by arrows)126to pass therethrough. As shown inFIG. 1, the light126passing through the transparent portion124originates from a portion128of an object130. The light126may originate from any scenery or objects located near the display screen102. In this example, the object130may be a viewer of the display screen102, and the image projected onto and captured by the image capture device104may include the viewer and any other scenery behind or near the viewer. The light126reaches the electronic image sensor of the image capture device104where the light projects an image thereon of the object portion128.

In an example, different portions of the display screen102may be controlled to be inactive in the sequence. As a result, the different portions of the display screen102may become at least partially transparent for allowing light to pass through different portions of the display screen102. In this way, different portions of the image of the viewer, other objects, and/or any other scenery behind or near the viewer are sequentially projected onto the image capture device104. The image capture device104may capture each image in the sequence and store the images in a memory132. As described in more detail herein, the captured images may be stitched together for generating a composite image of the viewer, other objects, and/or any other scenery. Further, several composite images may be generated based on images collected over a period of time for generating a video of the viewer, other objects, and/or any other scenery.

FIGS. 3A-3Cillustrate a front view of a display side110of a display screen102displaying images in a sequence and including partially transparent portions124that are at least partially transparent in accordance with embodiments of the present invention. InFIGS. 3A-3C, the first image displayed in the sequence begins withFIG. 3Aand continues in order to the last image shown inFIG. 3C. This example sequence of images is a video of a scene including a tree and surrounding scenery. In each figure, portion300corresponds to an area of the display screen102where a different set of light emitting components is activated to display the image of the scene in the sequence. The transparent portion124in each figure corresponds to a portion of the display screen102where the light emitting components are inactive for allowing light to pass through the display screen102and onto an image capture device positioned behind the display screen102. The light passing through the display screen102can project a sequence of images on the image capture device. The image capture device may capture the images. For example, the projected images may include an image of the portion128of the object130shown inFIG. 1. The projected images may also include images of other portions of the object130and/or any nearby objects or scenery. As illustrated, the position of the transparent portion124on the display screen102is different in each figure such that images of different portions of the object130and/or any nearby objects or scenery are projected onto and captured by the image capture device104. As described in further detail herein, the different captured images of an object and/or scenery surrounding the front side of the display screen102can be stitched together to form a composite still image. In accordance with embodiments of present invention, multiple, composite still images can be sequenced together for generating a video of the object130and/or any nearby objects or scenery.

Returning to the method ofFIG. 2, the method includes using206the image capture device to capture the sequence of projected images. For example, an electronic image sensor of the image capture device104may capture images projected by light passing through the partially transparent portion124or other partially transparent portions of the display screen102. Continuing the aforementioned example ofFIG. 3, the image capture device104may be controlled to capture the images projected by light passing through the partially transparent portions124and to generate image data representative of the projected images. The image capture interface118may receive the image data and store the image data in memory132.

In accordance with embodiments of the present invention, the image capture device104may capture images displayed by the display screen102. For example, activated OLEDs of the display screen may display images on the front and back sides of the display screen. Images displayed on the back side of the display screen may be projected onto and captured by the image capture device. For example, the back side112of the display screen102may project light134from activated light emitting components onto the image capture device104. The displayed images projected by the light134may be captured in sequence along with the capture of images projected by light134passing through the partially transparent portion124. For example, the image capture device104may capture a sequence of image frames that each contains a displayed image and the image projected by the light passing through the partially transparent portion. The captured sequence of images may be stored in the memory132as image data.

Continuing the example ofFIGS. 3A-3C,FIGS. 4A-4Cillustrate a sequence of image frames captured by an image capture device and corresponding to the capture of images projected by the display screen shown inFIGS. 3A-3C, respectively, in accordance with embodiments of the present invention. Particularly, the image frames shown inFIGS. 4A-4Care those captured by the image capture device104while the display screen displays the images shown inFIGS. 3A-3C, respectively, and the images of the object130shown inFIG. 1are projected through the transparent portion124. In each image ofFIGS. 4A-4C, image portion400corresponds to the image projected onto the image capture device by light passing through the transparent portion124shown inFIGS. 3A-3C. Further, in each image ofFIGS. 4A-4C, image portion402corresponds to the image projected onto the image capture device by the image displayed by the display screen102. These images may be stored in the memory132and time stamped. The images may be captured by the image capture device at the same or similar rate as the display screen sequentially displays images in sequence or image frames of a video.

The method ofFIG. 2includes generating208a still image based on the captured sequence of images. For example, an image generator136of the system100shown inFIG. 1may retrieve the captured image frames illustratedFIGS. 4A-4Cfrom the memory132, and identify the image data in the frames that represents the images generated by the light that passed through the portions400. The image generator136may assign this identified image data for each image frame as a component for use in generating a composite still image. Further, the image generator136may stitch together the components to generate the composite still image. For example, the portions400may be parts of an image of the object130. In another example, the image data acquired from the projected image may be made into a video.

As an example,FIG. 5illustrates the composite still image consisting of the stitched-together portions400shown inFIGS. 4A-4C. Referring toFIG. 5, the still image shows an image500of the object130.

Although three transparent portions of the display screen are provided in the sequence ofFIGS. 3A-3Cand used in the example ofFIGS. 4A-4C, it is emphasized the controller114may control the generation of any number of fully transparent or at least partially transparent portions in a sequence for capturing an image of objects and/or scenery near the display screen. Further, the transparent portions in the sequence may be positioned anywhere on the display screen. Further, the transparent portions may be of any suitable shape or size. The positioning, sequencing, sizes, and shapes of transparent portions may be suitably implemented by controlling the deactivation of light emitting components of the display screen such that groups of the light emitting components are suitably switched off in accordance with a desired position, shape, and/or size of a transparent portion.

In accordance with embodiments of the present invention, generation of a still image based on a captured sequence of images may include determining which portions of an image captured by the image capture device are to be used as components for generating the composite still image. For example, referring toFIGS. 4A-4C, each of the captured images include the image portion400that corresponds to the image projected onto the image capture device by light passing through the transparent portion of the display screen. Each of the captured images may also include the image portion402that corresponds to the image projected onto the image capture device by the image displayed by the display screen102. The image generator136may identify the image portions400in each of the captured images for use in generating the still image. In accordance with embodiments of the present invention, the image generator136may identify the image portions400by comparing each captured image to the displayed image data. The image generator136may determine that the image portions400do not match the displayed image data. In response to determining that an image portion does not match the display image data, the image generator136may use that image portion for generating the still image. For example, the image generator136may stitch together the image portions that do not match the display image based on an ordering of the capture of the images and/or positioning of the image portion in the captured image. For example, the captured image portions may be stitched together in the same position on an image that the captured image portions were located in the originally-captured image.

In accordance with embodiments of the present invention, the image generator136may identify the image portions to be used in generating the still image based on an ordering of the capture of the images and expected positions of the images within the captured images at the time of capture. For example, referring toFIGS. 4A-4C, the image generator136may store coordinates of an expected position of an image portion400when the image is captured based on the positioning of the light emitting coordinates that are inactive at the time of capture. The image generator136may use the image portions at the expected position coordinates for generating the still image. For example, the image generator136may stitch together the image portions at the expected position coordinates. In an example, the captured image portions may be stitched together in the same position on an image that the captured image portions were in the originally captured image.

In accordance with embodiments of the present invention, an image capture controller may receive a set of instructions for controlling a display screen to display a sequence of images. For example, the controller114shown inFIG. 1may receive a set of instructions for controlling the display screen102to display a sequence of still images or video. Such instructions may include stored frames of still image data and video data in a suitable image file format, such as, but not limited to, PNG, JPEG, GIF, Exif, TIFF, RAW, BMP, PPM, PGM, PBM, PNM, WEBP, and MPEG. The stored data may identify portions of each image for display that may be inactive when the image is displayed on a display screen. For example, the stored data may identify coordinates of portions400shown inFIGS. 4A-4C. These instructions may be used for controlling light emitting components of a display screen to activate in a sequence for displaying the sequence of images, and for controlling other light emitting components of the display screen that are not activated for displaying the images to be inactive such that a portion of the display screen is at least partially transparent in accordance with embodiments of the present invention.

In cases when coordinates of such portions are not identified in the received set of instructions, the controller114may alter the set of instructions for identifying portions of at least some of the images such that portions of the display screen corresponding to the identified portions become inactive when the corresponding image in the set of instructions is displayed. For example, the identified portions to become inactive may be the portions that become transparent such that light passes therethrough in accordance with embodiments of the present invention. A portion of an image to become inactive may correspond to portions of the image considered to be uninteresting to a viewer of the image. For example, the controller114may avoid inactivating portions corresponding to focal objects, such as people, in an image. Rather, the controller114may inactivate portions corresponding to background scenery. Focal objects and background scenery in an image may be identified by suitable techniques.

In accordance with embodiments of the present invention, a dithering technique may be applied for selecting light emitting components to activate to display an image on a display screen. The light emitting components that are not selected for activation may be controlled to be inactive such that light may pass therethrough for projecting an image on the image capture device in accordance with embodiments of the present invention. The captured images of light passing through the transparent portions is the inverse dither sequence of the displayed image. The captured image data may be used to dither the display data. The result can be a display image that is dithered to optimize both the projection to the image capture device and the displayed image. In an example, dithering may be used to create the illusion of new colors and shades by varying the pattern of pixels. Particularly, for example, the illusion of shades of gray may be produced by varying patterns of black and white dots.

Alternative to the still images and video being received from system memory, video or images for presentation on a display screen may be received from an image or video source. Example image and video sources include, but not limited to, a set-top-box (STB), a DVD player, a personal video recorder (PVR) a computer, cable/satellite broadcast television, and the like. Such sources may not have portions that are identified to become inactive. In accordance with embodiments of the present invention, such portions may be dynamically identified by an image capture controller prior to controlling the display screen to display the image or video.

When not operating in the image capture and display mode, the system100may operate in a display-only mode or an image capture-only mode. In the display-only mode, the controller114may control the display screen102to display still images and video, and the image capture device104may be turned off or otherwise set in an inactive state. Further, in the display-only mode, the display screen102may display still image or video that are viewable on the front display side112. The memory132may store frames of still image data and video data. The controller116or another component of the system100may access the still image data and video data stored in the memory132for use in controlling the display screen102to display the still images and video.

In the image capture-only mode, the display screen102may be turned off to become at least partially transparent. Thereby, light may pass through the display screen102for projecting images on the image capture device104. The image capture device104may capture the projected images and store a representation of the images in the memory132.

In other modes, the display screen102and/or image capture device104may be turned off or otherwise set to an inactive state for power conservation.

In accordance with embodiments of the present invention, the light generated by the display screen may be used for illuminating an object for capture of an image of the object by the image capture device. For example, the object130may be illuminated by the display screen102. In this example, light reflecting from the object130may return towards the display screen102, pass through the partially transparent portions, and project onto the image capture device104for use in generating still images and video in accordance with embodiments of the present invention.

Aspects of the present invention are described below with reference to flowchart illustrations and/or diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. For example, aspects of the present invention are described with reference to the diagram ofFIG. 1and the flowchart ofFIG. 2. It will be understood that each block of the flowchart illustrations and/or diagrams, and combinations of blocks in the flowchart illustrations and/or diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or diagram block or blocks.