Patent ID: 12190553

DETAILED DESCRIPTION OF THE DRAWINGS

The present subject matter provides a computer-implemented lighting system for video recording and display100(the lighting system100). In the example of the lighting system100shown inFIG.1, the lighting system100is implemented in a video conference call between a first user101using a first user device103and a second user105using a second user device107. Although the first user device103and the second user device107are shown as smartphones, it is understood that the first user device103and the second user device107may be any of a smartphone, a tablet, a video call specific portable device, a personal computer, a laptop, an office-based teleconferencing system, etc., as will be understood by those skilled in the art based on the teachings provided herein.

FIG.2illustrates the various systems of the first user device103shown inFIG.1. It is contemplated that the second user device107may also be represented by the elements shown inFIG.2.

FIG.2is a schematic diagram illustrating an example of the first user device103that may be used in the system shown inFIG.1. In the example shown inFIG.2, the lighting system100runs as a video conferencing application embodied in video conferencing software164on the first user device103. As shown inFIG.2, the first user device103maybe a mobile device, such as a smartphone, running video conferencing software164to provide the functionality described herein. A user may install the video conferencing software164on the first user device103via Apple's App Store, the Android Market, etc. The first user device103may include a wireless communication subsystem120to communicate with the second user device107.

The first user device103may include a memory interface102, controllers109, such as one or more data processors, image processors and/or central processors, and a peripherals interface106. The memory interface102, the one or more controllers109and/or the peripherals interface106can be separate components or can be integrated in one or more integrated circuits. The various components in the first user device103can be coupled by one or more communication buses or signal lines, as will be recognized by those skilled in the art.

Sensors, devices, and additional subsystems can be coupled to the peripherals interface106to facilitate various functionalities. For example, a motion sensor108, a light sensor110, and positioning sensors112can be coupled to the peripherals interface106to facilitate orientation, lighting, and positioning functions. Other sensors114can also be connected to the peripherals interface106, such as a proximity sensor, a temperature sensor, a biometric sensor, or other sensing device, to facilitate related functionalities.

A camera subsystem116includes a physical camera (e.g., a charged coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) optical sensor) which can be utilized to facilitate camera functions, such as recording photographs and video clips. Modern smartphones and other mobile devices typically feature more than one physical camera operated by the camera subsystem116. Such cameras may be located on the front of the first user device103—the side of the device with a touch screen display134(e.g., front facing cameras118) or rear of the first user device103—the side opposite the touch screen display134(e.g., rear facing cameras119).

Communication functions can be facilitated through a network interface, such as one or more wireless communication subsystems120, which can include radio frequency receivers and transmitters and/or optical (e.g., infrared) receivers and transmitters. The specific design and implementation of the communication subsystem120can depend on the communication network(s) over which the first user device103is intended to operate. For example, the first user device103can include communication subsystems120designed to operate over a GSM network, a GPRS network, an EDGE network, a 2G network, a 3G network, a 4G network, a 5G network, a Wi-Fi or Imax network, and a Bluetooth network. In particular, the wireless communication subsystems120may include hosting protocols such that the user device20may be configured as a base station for other wireless devices.

An audio subsystem122can be coupled to a speaker124and a microphone126to facilitate voice-enabled functions, such as voice recognition, voice replication, digital recording, and telephony functions.

The I/O subsystem128may include a touch screen controller130and/or other input controller(s)132. The touch screen controller130can be coupled to a touch screen display134. The touch screen display134and touch screen controller130can, for example, detect contact and movement, or break thereof, using any of a plurality of touch sensitivity technologies, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen display134. The other input controller(s)132can be coupled to other input/control devices136, such as one or more buttons, rocker switches, thumb-wheel, infrared port, USB port, and/or a pointer device such as a stylus. The one or more buttons (not shown) can include an up/down button for volume control of the speaker124and/or the microphone126.

The memory interface102may be coupled to memory104. The memory104can include high-speed random access memory and/or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and/or flash memory (e.g., NAND, NOR). The memory104may store operating system instructions140, such as Darwin, RTXC, LINUX, UNIX, OS X, iOS, ANDROID, BLACKBERRY OS, BLACKBERRY 10, WINDOWS, or an embedded operating system such as VxWorks. The operating system instructions140may include instructions for handling basic system services and for performing hardware dependent tasks. In some implementations, the operating system instructions140can be a kernel (e.g., UNIX kernel).

The memory104may also store communication instructions142to facilitate communicating with one or more additional devices, one or more computers and/or one or more servers. The memory104may include graphical user interface instructions144to facilitate graphic user interface processing; sensor processing instructions146to facilitate sensor-related processing and functions; phone instructions148to facilitate phone-related processes and functions; electronic messaging instructions150to facilitate electronic-messaging related processes and functions; web browsing instructions152to facilitate web browsing-related processes and functions; media processing instructions154to facilitate media processing-related processes and functions; GPS/Navigation instructions156to facilitate GPS and navigation-related processes and instructions; camera instructions158to facilitate camera-related processes and functions; and/or other software instructions160to facilitate other processes and functions (e.g., access control management functions, etc.). The memory104may also store other software instructions controlling other processes and functions of the user device20as will be recognized by those skilled in the art. In some implementations, the media processing instructions154are divided into audio processing instructions and video processing instructions to facilitate audio processing-related processes and functions and video processing-related processes and functions, respectively. An activation record and International Mobile Equipment Identity (IMEI)162or similar hardware identifier can also be stored in memory104. As described above, the video conferencing software164is also stored in the memory104and run by the controllers109.

Each of the above identified instructions and applications can correspond to a set of instructions for performing one or more functions described herein. These instructions need not be implemented as separate software programs, procedures, or modules. The memory104can include additional instructions or fewer instructions. Furthermore, various functions of the first user device103may be implemented in hardware and/or in software, including in one or more signal processing and/or application specific integrated circuits. Accordingly, the first user device103, as shown inFIG.2, may be adapted to perform any combination of the functionality described herein.

Aspects of the systems and methods described herein are controlled by one or more controllers109. The one or more controllers109may be adapted run a variety of application programs, access and store data, including accessing and storing data in associated databases, and enable one or more interactions via the first user device103. Typically, the one or more controllers109are implemented by one or more programmable data processing devices. The hardware elements, operating systems, and programming languages of such devices are conventional in nature, and it is presumed that those skilled in the art are adequately familiar therewith.

For example, the one or more controllers109may be a central control processing system utilizing a central processing unit (CPU), memory104and an interconnect bus. The CPU may contain a single microprocessor, or it may contain a plurality of microcontrollers109for configuring the CPU as a multi-processor system. The memory104include a main memory, such as a dynamic random access memory (DRAM) and cache, as well as a read only memory, such as a PROM, EPROM, FLASH-EPROM, or the like. The system may also include any form of volatile or non-volatile memory104. In operation, the main memory is non-transitory and stores at least portions of instructions for execution by the CPU and data for processing in accord with the executed instructions.

The one or more controllers109may further include appropriate input/output ports for interconnection with one or more output displays (e.g., monitors, printers, touch screen displays134, motion-sensing input device108, etc.) and one or more input mechanisms (e.g., keyboard, mouse, voice, touch, bioelectric devices, magnetic reader, RFID reader, barcode reader, touchscreen134, motion-sensing input device108, etc.) serving as one or more user interfaces for the processor. For example, the one or more controllers109may include a graphics subsystem to drive the output display (e.g., touch screen display134). The links of the peripherals to the system may be wired connections or use wireless communications.

Although summarized above as a smartphone-type implementation, those skilled in the art will recognize that the one or more controllers109also encompasses systems such as other mobile devices, host computers, servers, workstations, network terminals, PCs, and the like. In fact, the use of the term controller is intended to represent a broad category of components that are well known in the art.

A first example illustrating an application of the lighting system100shown inFIG.3includes a first user101and a second user105participating in a video conference with each other using their smartphones103and107. In this example, each user views a live video feed of the other user on a front facing display166(e.g., a touch screen display134) while simultaneously capturing a live video feed of themselves using the front facing camera118.

The lighting system100shown inFIG.3includes a first set of lighting elements168displayed as foreground lighting elements168. As shown inFIG.3, the first user device103includes a front facing camera118and a front facing display166. When running the video conferencing software164, the front facing display166provides a main display element170and a picture-in-picture display element172. The main display element170displays a first live video feed featuring the second user105captured by the second user device107and the picture-in-picture display element172displays a second live video feed featuring the first user101captured by the first user device103.

Similarly, the second user device107includes a front facing camera118and a front facing display166. When running the video conferencing software164, the front facing display166provides a main display element170and a picture-in-picture display element172. The main display element170displays the second live video feed featuring the first user101captured by the first user device103and the picture-in-picture display element172displays the first live video feed featuring the second user105captured by the second user device107.

In the example shown inFIG.3, the first set of lighting elements168are displayed as foreground lighting elements168lining the perimeter of the front facing display166. The lighting elements168shown are bright, white pixels that illuminate the face of the first user101to improve the lighting of the second live video being captured by the first user device103. The lighting elements168in this example are shown as foreground lighting elements168, meaning they are displayed in front of the videos shown on the main display element170and the picture-in-picture display element172.

The example shown inFIG.4is similar to the example shown inFIG.3, but adds an illumination indicator174and also displays the first set of lighting elements168in the picture-in-picture display element172on the second user device107. Accordingly, the second user105is able to see that the first user device103is using a first set of lighting elements168to illuminate the first user101and can additionally see what the second user105looks like on the first user device103in combination with the first set of lighting elements168.

The example shown inFIG.5is similar to the example shown inFIG.4, but the first set of lighting elements168are shown as background lighting elements168on the first user device103. In other words, the lighting elements168disappear behind the video of the second user105. The lighting elements168can be displayed as background lighting elements168when one of the first user device103and the second user device107segment the second user105from the background176in the first live video feed.

The example shown inFIG.6is similar to the example shown inFIG.1, but the foreground lighting elements168are displayed on both of the first user device103and the second user device107.

In the example shown inFIG.7, the second user105is segmented from the background176of the first video and the entirety of the background176is replaced with background lighting elements168on the second user device107.

FIGS.8-16illustrate various examples of lighting elements168in various configurations and geometries. The varied examples shown in the drawings illustrate how the lighting elements168may be foreground lighting elements168, may be background lighting elements168, may replace segmented backgrounds176, may be provided on one or both user devices simultaneously, etc.

Turning now toFIG.17, an example is provided in which a first user101uses a first user device103including a front facing camera118and a front facing display166to take a selfie photograph or record a selfie-style video. Just as shown in the example provided with reference toFIG.3, the first set of lighting elements168are displayed as foreground lighting elements168lining the perimeter of the front facing display166. The lighting elements168shown are bright, white pixels that illuminate the face of the first user101to improve the lighting of the content to be captured by the first user device103.

FIG.18illustrates an example of lighting activation options178presented to the user101when taking a photograph or recording a video using a front facing camera118. In the example shown inFIG.18, when the first user device103identifies that the ambient lighting conditions are poor for capturing a well-lit photograph or video, the front facing display166presents a notification180to the user101stating “Poor Lighting Conditions Detected.” The display166further presents three lighting element activation options178for activating lighting, including “yes,” “auto,” and “no.” In this example, the options are provided with respect to background lighting elements168, though the concepts described herein apply equally to foreground and other lighting elements168. When the user101selects the “yes” option, the lighting elements168are activated regardless of the ambient lighting conditions. When the user101selects the “auto” option, the lighting elements168are activated when the lighting system100determines that the lighting quality of the content to be captured will be improved by activating the lighting elements168. When the user101selects the “no” option, the lighting elements168are not activated regardless of the ambient lighting conditions.

FIG.19illustrates an example of using lighting elements168to replace a segmented background176when taking a photograph or recording a video using a front facing camera118. As shown in the example provided inFIG.19, the display166presents a notification180that the a “Background Lighting Activated” mode is active. In this mode, a segmented background176in the main display element170is replaced by lighting elements168. As further shown inFIG.19, the display166provides a picture-in-picture display element172that provides the user101a representation of the content to be captured. As shown, the representation in the picture-in-picture display element172does not include the lighting elements168, but instead shows the background that will be captured by the front facing camera118. In this embodiment, the user101is able to see the content that is to be captured while still benefiting from the use of the lighting elements168.

FIG.20illustrates an example of a picture-in-picture display element172that shows the effects of the lighting system100when taking a photograph or recording a video using a front facing camera118. As shown, the picture-in-picture display element172presents the user with a representation of the content that would be captured under the lighting conditions in the absence of the activation of the lighting elements168, which in this example are shown as segmented background lighting elements168. This presentation enables the user101to determine whether the lighting effects168are necessary to provide the appropriate lighting conditions. As further shown inFIG.20, the display166presents the user101with an illumination indicator174to indicate that the lighting elements168are active.

It should be noted that various changes and modifications to the presently preferred embodiments described herein will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the present invention and without diminishing its attendant advantages.