Source: https://patents.google.com/patent/US10432872B2/en
Timestamp: 2020-08-08 15:47:12
Document Index: 383618852

Matched Legal Cases: ['Application No. 62', 'Application No. 62', 'Application No. 62', 'Application No. 62', 'Application No. 105135139', 'Application No. 105134986', 'Application No. 105134992', 'Application No. 105134986', 'Application No. 105134992', 'Application No. 105135139']

US10432872B2 - Mobile device with display overlaid with at least a light sensor - Google Patents
US10432872B2
US10432872B2 US15/442,576 US201715442576A US10432872B2 US 10432872 B2 US10432872 B2 US 10432872B2 US 201715442576 A US201715442576 A US 201715442576A US 10432872 B2 US10432872 B2 US 10432872B2
US15/442,576
US20170171448A1 (en
2017-02-24 Assigned to Essential Products, Inc. reassignment Essential Products, Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: EVANS, DAVID JOHN, GAGNE-KEATS, JASON SEAN, HERSHENSON, MATTHEW, JIANG, XINRUI, MIAO, XIAOYU, RUBIN, ANDREW E., TATE, JOSEPH ANTHONY, ZAVIN, REBECCA SCHULTZ
2017-02-24 Priority to US15/442,576 priority patent/US10432872B2/en
2017-02-24 Application filed by Essential Products Inc filed Critical Essential Products Inc
2017-06-15 Publication of US20170171448A1 publication Critical patent/US20170171448A1/en
2017-08-30 Assigned to Essential Products, Inc. reassignment Essential Products, Inc. CORRECTIVE ASSIGNMENT TO CORRECT THE FIRST NAMED INVENTOR NAME PREVIOUSLY RECORDED AT REEL: 041377 FRAME: 0459. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Assignors: EVANS, DAVID JOHN, V, GAGNE-KEATS, JASON SEAN, HERSHENSON, MATTHEW, JIANG, XINRUI, MIAO, XIAOYU, RUBIN, ANDREW E., TATE, JOSEPH ANTHONY, ZAVIN, REBECCA SCHULTZ
2019-10-01 Publication of US10432872B2 publication Critical patent/US10432872B2/en
230000000007 visual effect Effects 0.000 claims description 9
239000004973 liquid crystal related substances Substances 0.000 claims description 8
239000010410 layers Substances 0.000 description 80
230000003667 anti-reflective Effects 0.000 description 1
This application is a continuation of U.S. Utility application Ser. No. 15/283,112, filed on Sep. 30, 2016. This application incorporates by reference U.S. Provisional Patent Application No. 62/249,130, filed on Oct. 30, 2015, U.S. Provisional Application No. 62/300,631, filed on Feb. 26, 2016, U.S. Provisional Application No. 62/319,099, filed on Apr. 6, 2016, and U.S. Provisional Application No. 62/373,910, filed on Aug. 11, 2016, all of which are incorporated by reference herein in their entirety.
displaying a user interface on at least a portion of the display over the light sensor, the display comprising a plurality of pixels, wherein opaqueness of each pixel in the plurality of pixels is individually adjustable;
responsive to the command, adjusting, by the processor, opaqueness of at least a target portion of an opaqueness adjustable region of the display directly over the light sensor, wherein the opaqueness of each pixel in the opaqueness adjustable region is individually adjustable creating a visual effect comprising a grating or a lens flare;
based on the visual effect, determining, by the processor, a gesture from the image; and
further responsive to the command, the processor capturing the image with the light sensor while the target portion of the opaqueness adjustable region is at least partially transparent.
2. The computer-implemented method of claim 1, wherein adjusting the opaqueness includes turning at least the target portion substantially transparent.
3. The computer-implemented method of claim 1, wherein opaqueness and transparency of the opaqueness adjustable region correspond to detectability of external light outside of the mobile device by the light sensor.
4. The computer-implemented method of claim 1, wherein the display is an organic light emitting diode (OLED) display that is substantially transparent when light emitting pixels of OLED are turned off, and wherein said adjusting the opaqueness includes turning off light emitting pixels of the OLED at the target portion to make the target portion substantially transparent.
5. The computer-implemented method of claim 4, wherein the target portion of the OLED display is turned off whenever the light sensor is active.
6. The computer-implemented method of claim 1, wherein the display is a liquid crystal display (LCD), and wherein said adjusting the opaqueness includes causing one or more liquid crystals of the LCD at the target portion to optically rotate light passing through the LCD toward the light sensor.
7. The computer-implemented method of claim 1, further comprising detecting a touch event utilizing a touch sensor integrated with the display of the mobile device, and wherein said adjusting the opaqueness is responsive to detecting the touch event.
8. The computer-implemented method of claim 1, wherein the light sensor is part of a video camera, a still image camera, a proximity sensor system, an optical biometric sensor system, an optical gesture detection system, or any combination thereof.
9. The computer-implemented method of claim 1, wherein said capturing the image is performed while the mobile device is displaying a video on the display.
analyzing the image to identify the gesture; and
providing the gesture to an operating system of the mobile device to trigger a command associated with the gesture.
11. The computer-implemented method of claim 1, wherein the target portion of the opaqueness adjustable region is kept substantially transparent for a duration sufficiently short to be substantially undetectable by human perception.
12. The computer-implemented method of claim 1, wherein said adjusting the opaqueness includes keeping the target portion substantially transparent for substantially a duration needed to capture the image according to the command.
13. The computer-implemented method of claim 1, further comprising displaying an opaque portion within the opaqueness adjustable region of the display while the target portion of the opaqueness adjustable region is rendered substantially transparent.
14. The computer-implemented method of claim 1, further comprising turning at least the target portion substantially translucent.
15. The computer-implemented method of claim 14, wherein the captured image is a first image; and further comprising displaying a second image over the target portion while capturing the first image.
utilizing the captured image to recognize a biometric pattern; and
providing access to at least a subset of the user interface based on the recognized biometric pattern.
utilizing the captured image to videoconference with an external device; and
wherein the light sensor is positioned in the middle of the display to enable a user of the mobile device to maintain eye contact with an external user of the external device while videoconferencing.
identifying a real world object from the captured image; and
rendering, in substantially real-time relative to said capturing the image, a video representation of the real world object on the display of the mobile device.
19. The computer-implemented method of claim 18, wherein said capturing occurred between two consecutive frames of the video representation when the target portion of the opaqueness adjustable region of the display is substantially transparent.
20. The computer-implemented method of claim 1, further comprising flashing a light source from underneath the display immediately before or during said capturing of the image.
21. The computer-implemented method of claim 1, further comprising flashing the portion of the display immediately before or during said capturing of the image.
22. A computer readable data storage memory storing computer-executable instructions that, when executed by a computer system, cause the computer system to perform a computer-implemented method, the computer-executable instructions comprising:
instructions for identifying a command to capture an image with a light sensor of a mobile device;
instructions for, responsive to the command, adjusting opaqueness of at least a target portion of an opaqueness adjustable region of a display directly over the light sensor from substantially opaque to substantially transparent, the display comprising a plurality of pixels, wherein opaqueness of each pixel in the plurality of pixels is individually adjustable, wherein the opaqueness of each pixel in the opaqueness adjustable region is individually adjustable creating a visual effect comprising a grating or a lens flare;
instructions for determining a gesture from the image based on the visual effect; and
instructions for, further responsive to the command, capturing the image with the light sensor while the target portion of the opaqueness adjustable region is substantially transparent.
a transparent screen comprising a plurality of pixels, wherein opaqueness of each pixel in the plurality of pixels is individually adjustable;
an opaqueness adjustable display underneath the transparent screen, wherein the opaqueness adjustable display is capable of turning at least a region thereon substantially transparent, wherein the opaqueness of each pixel in the opaqueness adjustable display is individually adjustable creating a visual effect comprising a grating or a lens flare;
a processor to determine a gesture from an image based on the visual effect; and
a camera disposed underneath the transparent screen and at least a portion of the opaqueness adjustable display, wherein the camera is configured to capture the image while a target portion of the opaqueness adjustable display is substantially transparent.
24. The mobile device of claim 23, wherein the opaqueness adjustable display is an organic light emitting diode (OLED) display or a liquid crystal display (LCD).
25. The mobile device of claim 23, further comprising a touch sensing panel coupled to the transparent screen, wherein the touch sensing panel is substantially transparent.
displaying a user interface on at least a portion of a display over a light sensor, the display comprising a plurality of pixels, wherein opaqueness of each pixel in the plurality of pixels is individually adjustable;
US15/442,576 2015-10-30 2017-02-24 Mobile device with display overlaid with at least a light sensor Active US10432872B2 (en)
US15/442,576 US10432872B2 (en) 2015-10-30 2017-02-24 Mobile device with display overlaid with at least a light sensor
US15/283,112 Continuation US9754526B2 (en) 2015-10-30 2016-09-30 Mobile device with display overlaid with at least a light sensor
US20170171448A1 US20170171448A1 (en) 2017-06-15
US10432872B2 true US10432872B2 (en) 2019-10-01
US15/442,576 Active US10432872B2 (en) 2015-10-30 2017-02-24 Mobile device with display overlaid with at least a light sensor
US (2) US9754526B2 (en)
US10229316B2 (en) * 2016-01-29 2019-03-12 Synaptics Incorporated Compound collimating system using apertures and collimators
CN106878564A (en) * 2017-03-07 2017-06-20 广东欧珀移动通信有限公司 A kind of control method and device of display screen, display screen and mobile terminal
US10681281B2 (en) 2017-12-28 2020-06-09 Htc Corporation Mobile device, and image processing method for mobile device
CN109348012B (en) * 2018-11-16 2020-04-03 Oppo广东移动通信有限公司 Electronic device, control method thereof and control device thereof
CN109327577B (en) * 2018-11-16 2020-02-11 Oppo广东移动通信有限公司 Electronic device, control method thereof and control device thereof
TW445744B (en) 1998-11-16 2001-07-11 Acer Inc Embedded CMOS camera in a laptop computer
US6992718B1 (en) 1998-08-31 2006-01-31 Matsushita Electric Industrial Co., Ltd. Illuminating apparatus, display panel, view finder, video display apparatus, and video camera mounting the elements
US20060146008A1 (en) 2003-06-26 2006-07-06 Johnson Mark T Method for calibrating an electrophoretic dispaly panel
CN101048727A (en) 2004-10-27 2007-10-03 富士通天株式会社 Display
US20080106629A1 (en) 2006-11-02 2008-05-08 Kurtz Andrew F integrated display having multiple capture devices
TW201001364A (en) 2008-03-28 2010-01-01 Qualcomm Mems Technologies Inc Apparatus and method of dual-mode display
US20100117949A1 (en) 2008-11-10 2010-05-13 Wistron Corp. Control method for backlight module of lcd and application thereof
US20100177179A1 (en) 2009-01-15 2010-07-15 International Business Machines Corporation Apparatus and method for enhancing field of vision of the visually impaired
US20110050657A1 (en) 2009-08-25 2011-03-03 Seiko Epson Corporation Electro-optical device and electronic apparatus
US20110285934A1 (en) 2009-02-06 2011-11-24 Sharp Kabushiki Kaisha Display device
TW201207535A (en) 2010-05-27 2012-02-16 E Ink Corp Dual mode electro-optic displays
CN102379002A (en) 2009-02-06 2012-03-14 全球Oled科技有限责任公司 Light sensing in display device
US20120068913A1 (en) * 2010-09-21 2012-03-22 Avi Bar-Zeev Opacity filter for see-through head mounted display
US20130286152A1 (en) 2012-04-26 2013-10-31 Sony Mobile Communications Ab Screen camera
US20130321686A1 (en) 2012-06-01 2013-12-05 Kar-Han Tan Display-camera system with switchable diffuser
TWM480723U (en) 2014-01-06 2014-06-21 Superc Touch Corp High-accuracy in-cell active matrix OLED display touch structure of narrow border
US20140253775A1 (en) 2008-11-28 2014-09-11 Lg Electronics Inc. Control of input/output through touch
US20150035936A1 (en) 2013-07-31 2015-02-05 Hewlett-Packard Development Company,L.P. Display-camera system
US20150062052A1 (en) * 2012-05-09 2015-03-05 Apple Inc. Device, Method, and Graphical User Interface for Transitioning Between Display States in Response to a Gesture
TW201523460A (en) 2013-12-10 2015-06-16 Htc Corp Method used in interactive device and for recognizing user behavior operating on the interactive device, and associated interactive device and computer readable medium
WO2015097490A1 (en) 2013-12-27 2015-07-02 Intel Corporation Display having an integrated camera
US20160206207A1 (en) * 2015-01-19 2016-07-21 Samsung Electronics Company, Ltd. Optical Detection and Analysis of Internal Body Tissues
CN106019671A (en) 2016-07-22 2016-10-12 杨永建 Unit integrating display and camera shooting, component and panel
US20170084231A1 (en) * 2015-09-23 2017-03-23 Intel Corporation Imaging system management for camera mounted behind transparent display
US20170124932A1 (en) 2015-10-30 2017-05-04 Essential Products, Inc. Mobile device with display overlaid with at least a light sensor
US20180013944A1 (en) 2016-02-26 2018-01-11 Essential Products, Inc. Image capture with a camera integrated display
2016-10-28 TW TW105134991A patent/TWI623864B/en not_active IP Right Cessation
2017-02-24 US US15/442,576 patent/US10432872B2/en active Active
US20180107241A1 (en) 2015-10-30 2018-04-19 Essential Products, Inc. Camera integrated into a display
US20170330507A1 (en) 2015-10-30 2017-11-16 Essential Products, Inc. Light sensor beneath a dual-mode display
Advisory Action dated May 12, 2017, for U.S. Appl. No. 15/231,643 of Evans et al. filed Aug. 8, 2016, 4 pages.
Corrected Notice of Allowability dated Nov. 16, 2017 for U.S. Appl. No. 15/444,320 of Evans et al., filed Feb. 27, 2017.
Decision to Grant dated Mar. 6, 2018 for Taiwan Application No. 105135139 of Essential Products, Inc.
Evans V , et al., "Camera Integrated Into a Display", Non-Final Office Action mailed in U.S. Appl. No. 15/231,664 dated Feb. 16, 2017; 27 pages.
Evans V , et al., International Search Report and Written Opinion received in PCT Application No. PCT/US16/59524, dated Feb. 27, 2017; 10 pages.
Evans V , et al., Non-Final Office Action mailed in U.S. Appl. No. 15/336,380 dated Mar. 14, 2017; 17 pages.
Evans V , et al., Non-Final Office Action received in U.S. Appl. No. 15/231,652 dated Feb. 24, 2017; 26 pages.
Final Office Action dated Apr. 11, 2017 in U.S. Appl. No. 15/231,664 of Evans et al. filed Aug. 8, 2016, 36 pages.
Final Office Action dated Apr. 18, 2017 in U.S. Appl. No. 15/283,204 of Evans et al. filed Sep. 30, 2016, 17 pages.
Final Office Action dated Apr. 26, 2017 in U.S. Appl. No. 15/231,652 of Evans et al. filed Aug. 8, 2016, 32 pages.
Final Office Action dated Mar. 22, 2018 for U.S. Appl. No. 15/283,204 of Evans et al. filed Sep. 30, 2016.
Final Office Action dated Mar. 31, 2017 in U.S. Appl. No. 15/231,643 of Evans et al. filed Aug. 8, 2016, 32 pages.
International Search Report and Written Opinion in PCT/US16/58543 dated Jan. 3, 2017, 12 pages.
International Search Report and Written Opinion in PCT/US16/58545 dated Jan. 3, 2017, 12 pages.
International Search Report and Written Opinion in PCT/US16/58548 dated Jan. 19, 2017, 8 pages.
International Search Report and Written Opinion in PCT/US16/58947 dated Jan. 13, 2017, 9 pages.
International Search Report and Written Opinion in PCT/US16/59550 dated Jan. 3, 2017, 10 pages.
Int'l Preliminary Report on Patentability dated May 11, 2018 for PCT Patent Application No. PCT/US2016/058543 of Essential Products, Inc.
Int'l Preliminary Report on Patentability dated May 11, 2018 for PCT Patent Application No. PCT/US2016/058545 of Essential Products, Inc.
Int'l Preliminary Report on Patentability dated May 11, 2018 for PCT Patent Application No. PCT/US2016/059550 of Essential Products, Inc.
Non-Final Office Action dated Dec. 15, 2016 in U.S. Appl. No. 15/283,112 of Evans et al. filed Sep. 30, 2016, 26 pages.
Non-Final Office Action dated Dec. 2, 2016 in U.S. Appl. No. 15/283,204 of Evans et al. filed Sep. 30, 2016, 12 pages.
Non-Final Office Action dated Feb. 23, 2018 for Taiwan Application No. 105134986 of Essential Products, Inc.
Non-Final Office Action dated Jan. 5, 2017 in U.S. Appl. No. 15/231,643 of Evans et al. filed Aug. 8, 2016, 37 pages.
Non-Final Office Action dated Jun. 20, 2018 for U.S. Appl. No. 15/712,034 of Evans et al.
Non-Final Office Action dated Jun. 23, 2017 in U.S. Appl. No. 15/231,664 of Evans et al.
Non-Final Office Action dated Nov. 9, 2017 for U.S. Appl. No. 15/665,359 of Evans et al., filed Jul. 31, 2017.
Non-Final Office Action dated Sep. 19, 2017 for U.S. Appl. No. 15/283,204 of Evans et al. filed Sep. 30, 2016.
Non-Final Office Action dated Sep. 21, 2017 for TW Application No. 105134992 of Evans et al., filed Oct. 28, 2016.
Notice of Allowance dated Apr. 25, 2018 in U.S. Appl. No. 15/665,359 for Evans V, D.J., filed Jul. 31, 2017.
Notice of Allowance dated Aug. 30, 2017 for U.S. Appl. No. 15/231,643 of Evans et al., filed Oct. 30, 2016.
Notice of Allowance dated Jul. 14, 2017 in U.S. Appl. No. 15/231,652 of Evans et al. filed Aug. 8, 2016.
Notice of Allowance dated Jun. 4, 2018 for Taiwan Patent Application No. 105134986 of Essential Products, Inc.
Notice of Allowance dated Jun. 4, 2018 for U.S. Appl. No. 15/283,204 of Evans et al.
Notice of Allowance dated May 19, 2017, for U.S. Appl. No. 15/283,112 of Evans et al. filed Sep. 30, 2016, 8 pages.
Notice of Allowance dated May 26, 2017, for U.S. Appl. No. 15/336,380 of Evans, J.D. et al. filed on Oct. 27, 2017.
Notive of Allowance dated Sep. 13, 2017 for U.S. Appl. No. 15/231,664 of Evans et al. filed Aug. 8, 2016.
Office Action dated Feb. 13, 2018 for Taiwan Patent Application No. 105134992 of Essential Products, Inc.
Office Action dated Jan. 25, 2018 for Taiwan Patent Application No. 105135139 of Essential Products, Inc.
Supplemental Notice of Allowance dated Sep. 19, 2017 for U.S. Appl. No. 15/444,320 of Evans et al., filed Feb. 27, 2017.
U.S. Appl. No. 15/444,320 of Evans et al. filed Feb. 27, 2017, 82 pages.
US9754526B2 (en) 2017-09-05
ES2716613T3 (en) 2019-06-13 Display device configured as lighting source
KR20190091557A (en) 2019-08-06 Display screen assembly, electronics and how to acquire an image
US20170154570A1 (en) 2017-06-01 Organic light emitting diode (oled) display apparatus having light sensing function
KR102051656B1 (en) 2019-12-03 Transparent display apparatus and method thereof
JP6596205B2 (en) 2019-10-23 Information processing device
US8654243B2 (en) 2014-02-18 Image pickup apparatus and control method thereof
US9741286B2 (en) 2017-08-22 Interactive display panel with emitting and sensing diodes
KR102022553B1 (en) 2019-09-19 Hmd apparatus including light turning element
WO2017129126A1 (en) 2017-08-03 Under-screen optical sensor module for on-screen fingerprint sensing
TWI586167B (en) 2017-06-01 Controlling a camera with face detection
US9866825B2 (en) 2018-01-09 Multi-view image display apparatus and control method thereof
US9176625B2 (en) 2015-11-03 Information input device, information input method, information input-output device, storage medium, and electronic unit
KR101900847B1 (en) 2018-09-20 Display device
US20170064291A1 (en) 2017-03-02 Display apparatus, head-mounted display apparatus, image display method, and image display system
US20150243079A1 (en) 2015-08-27 Head mounted display providing closed-view and method of controlling therefor
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:EVANS, DAVID JOHN;JIANG, XINRUI;RUBIN, ANDREW E.;AND OTHERS;REEL/FRAME:041377/0459
Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE FIRST NAMED INVENTOR NAME PREVIOUSLY RECORDED AT REEL: 041377 FRAME: 0459. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNORS:EVANS, DAVID JOHN, V;JIANG, XINRUI;RUBIN, ANDREW E.;AND OTHERS;REEL/FRAME:043723/0490
2019-06-18 STCB Information on status: application discontinuation