Patent ID: 12216277

DESCRIPTION OF EXAMPLE EMBODIMENTS

Images captured by an image sensor of an electronic device may be blurred for a variety of reasons. For example, a front-facing camera may be disposed under a display (or another surface) of a personal electronic device, for example to decrease the camera's footprint on the surface of the device and to increase the useable surface of the device. As an example, placing a front-facing camera behind a display of an electronic device may increase the size of the device's display. However, disposing the front-facing camera system behind the display panel may degrade images captured by the front-facing camera, for example because of interference caused by the display structure as light passes through the display to the camera sensor, as described more fully below.

One or more PSF measurements can be performed to characterize the degradation of an image captured by an image sensor, such as an under-display camera. Each PSF may be a function of the source's wavelength, distance, and angle with the optical axis with respect to the camera's sensor. The set of PSFs are then used to reconstruct an undegraded image via deconvolution, for example by convolving the blurred image with the inverse PSFs. The deconvolution is typically performed as a computational, numerical process by a computing device, such as by a processor of a computing device. Deconvolution calculations consume system resources, such as power, memory, and available processing capabilities, and for a device to perform deconvolution computationally the device must have the minimum capabilities, such as processing capabilities, necessary to perform the computations.

In contrast, embodiments of this disclosure perform deconvolution in the optical domain by using one or more physical optical elements designed to deblur images. The physical optical element optically performs deconvolution by manipulating the light that reaches a camera's sensor, and therefore requires little or no computation to be performed by the device in order to de-blur an image. Embodiments of this disclosure therefore reduce systems cost, computational requirements, power budget, and the time required for image reconstruction. Moreover, embodiments discloses herein enable deconvolution techniques to be performed by lightweight devices that do not have the computational resources necessary to perform deconvolution.

FIG.1illustrates an example system including a corrective optical element that performs deconvolution. As shown in the example ofFIG.1, source110, which may be a point source or may represent a point source, lies in an object plane and creates optical source field S. As explained more fully herein, source field S may be incoherent light comprised of many different wavelengths, e.g., wavelengths in the visible spectrum. Source field S arrives at, and is modified by, mask120, which for example represents the layers of a physical display screen, for example a display screen of a personal electronic device, such as a mobile phone, tablet computer, smartwatch, camera, laptop, monitor, television, and so forth As shown in the example ofFIG.1, source field S passes through a transparent portion of the display screen and may interact with one or more optical components130. For example, optical components130may include one or more lenses, which may be part of a camera system, that collect and focus light. While the example ofFIG.1illustrates a single optical component130, this disclosure contemplates that more than one optical component130may be present, and moreover that in some embodiments all or some optical components may be disposed prior to corrective mask140while in some embodiments all or some optical components may be disposed after corrective mask140.

As shown in the example ofFIG.1, after passing through mask120and optical component130, source field S reaches corrective mask140(also referred to herein as an “optical element” or “corrective optical element”). Corrective mask140interacts with and modifies source field S so that the optical field emerging from corrective element140is image field I. As explained more fully below, corrective mask140deconvolves the blurring introduced to source field S, e.g., by the display structure represented by mask120. The resulting image field I represents a de-blurred image field, and as shown in the example ofFIG.1, de-blurred image field I is captured by image sensor150, for example an image sensor of a camera. Thus, the resulting image captured by image sensor150need not be deconvolved using computational deconvolution techniques, because corrective mask140has physically performed the deconvolution necessary to deblur the image field so that the camera captures an already de-blurred image.

While the example ofFIG.1relates to deconvolving an image captured by a camera that is mounted under a display of a device, this disclosure contemplates that the optical element disclosed herein may perform deconvolution in any suitable device or context. Moreover, while the example ofFIG.1describes the camera and optical element—in the context of a personal electronic device with a display, this disclosure contemplates that the components described herein may be disposed in any suitable device that has an at least partially optically transparent surface, such as for example an appliance, mirror, etc.

FIG.2illustrates an example method for creating, or fabricating, an optical element that performs deconvolution, such as corrective mask140of the example ofFIG.1. The example method ofFIG.2may begin at step210, which includes propagating, for each of one or more wavelengths λi, a point source field from an object plane to a corrective mask plane to determine a source field Si. For example, as explained above with reference toFIG.1, point source110lies in an object plane, while the corrective mask plane is the plane in which corrective mask140lies. This disclosure contemplates that any suitable number of wavelengths λimay be used to generate corresponding source fields Si. For example, a set of wavelengths across the visible spectrum may be chosen, each wavelength corresponding to a source field Si.

At step220, the method ofFIG.2includes propagating, for each of the one or more wavelengths λi, the point source field Sifrom an image plane to the corrective mask plane to determine an image field Ii. For example, with reference toFIG.1, the image plane corresponds to the plane in which sensor150is disposed. At step230, the method ofFIG.2includes determining, for each of the one or more wavelengths λi, a phase modulation field Φibased on the source field Siand the image field Ii. For example, each phase modulation field Φimay be equal to

SiIi.

At step240, the method ofFIG.2includes determining a multi-wavelength phase modulation field Φ based on combining the phase modulation field for each of the one or more wavelengths λi. The combination of each Φiinto the multi-wavelength phase modulation field Φ may be accomplished using any suitable techniques, such as for example using superposition (complex addition), random sampling, etc.

Particular embodiments may repeat one or more steps of the method ofFIG.2, where appropriate. Although this disclosure describes and illustrates particular steps of the method ofFIG.2as occurring in a particular order, this disclosure contemplates any suitable steps of the method ofFIG.2occurring in any suitable order. Moreover, this disclosure contemplates that some or all of the computing operations described herein, including certain steps of the example method illustrated inFIG.2such as steps230and240, may be performed by circuitry of a computing device described herein, by a processor coupled to non-transitory computer readable storage media, or any suitable combination thereof.

The multi-wavelength phase modulation field Φ can be used to design and fabricate a physical corrective optical element, which in particular embodiments may occur after simulation studies are used to test and validate Φ. A physical corrective optical element that deblurs distortion caused by, e.g., the layers of a device's display can be physically generated from the multi-wavelength phase modulation field Φ by any of a variety of different fabrication methods known in the field, including, for example binary amplitude masks, phase masks, kinoforms, freeform holographic optical elements, metalenses, etc.

Once the physical optical element is fabricated from the multi-wavelength phase modulation field Φ, the optical element can be disposed in a device at the location of the corrective mask plane described above. The device can then perform image deconvolution using the fabricated optical element, without having to deconvolve the images computationally.

This disclosure contemplates that a corrective optical element may be disposed in any suitable location in a device. For example,FIG.1illustrates an example in which the corrective optical element (correction mask140) is disposed between a display structure (mask120) and an image sensor150.FIG.3illustrates another example where the corrective mask340is combined (i.e., co-located) with the display mask320to form a combined mask. For example, a corrective mask may be placed on the bottom of the display layers of a device.

Moreover, while examples in this disclosure relate to deconvolution and image capture of visible light, this disclosure contemplates that the optical elements described herein may be applied to other spectrums. For example, a depth sensor may sense electromagnetic waves in a spectrum that includes UV radiation or infrared radiation, or both, and an optical element may be fabricated for the depth sensor by, for example, including wavelengths in those spectrums in steps210and220of the example method ofFIG.2.

FIG.4illustrates an example optical response of a system using a corrective optical element as described herein.FIG.4illustrates an example display structure mask410, with the dark regions indicting non-transparent features of the display mask and the white regions indicating transparent (or partially transparent) regions. As a result of mask410, light from a point source passing through mask410has a PSF as shown in image420, which illustrates the blurring that display structure410introduces.

FIG.4also illustrates an example of a corrective phase mask440, where phase modulation is partly represented by the intensity of the greyscale shown. A physical optical element can be fabricated from phase mask440and introduced to correct the blurring shown in image420. For example, physical mask450illustrates an example where a binarized phase mask based on physical mask440is combined with the physical structure of display mask430. The PSF of display mask450is shown in image460, which illustrates deblurring of a point source relative to image420for display mask410.

FIG.5illustrates an example computer system500. In particular embodiments, one or more computer systems500perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems500provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems500performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems500. Herein, reference to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, reference to a computer system may encompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems500. This disclosure contemplates computer system500taking any suitable physical form. As example and not by way of limitation, computer system500may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (such as, for example, a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile telephone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system500may include one or more computer systems500; be unitary or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems500may perform without substantial spatial or temporal limitation one or more steps of one or more methods described or illustrated herein. As an example and not by way of limitation, one or more computer systems500may perform in real time or in batch mode one or more steps of one or more methods described or illustrated herein. One or more computer systems500may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein, where appropriate.

In particular embodiments, computer system500includes a processor502, memory504, storage506, an input/output (I/O) interface508, a communication interface510, and a bus512. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor502includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor502may retrieve (or fetch) the instructions from an internal register, an internal cache, memory504, or storage506; decode and execute them; and then write one or more results to an internal register, an internal cache, memory504, or storage506. In particular embodiments, processor502may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor502including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor502may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory504or storage506, and the instruction caches may speed up retrieval of those instructions by processor502. Data in the data caches may be copies of data in memory504or storage506for instructions executing at processor502to operate on; the results of previous instructions executed at processor502for access by subsequent instructions executing at processor502or for writing to memory504or storage506; or other suitable data. The data caches may speed up read or write operations by processor502. The TLBs may speed up virtual-address translation for processor502. In particular embodiments, processor502may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor502including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor502may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors502. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

In particular embodiments, memory504includes main memory for storing instructions for processor502to execute or data for processor502to operate on. As an example and not by way of limitation, computer system500may load instructions from storage506or another source (such as, for example, another computer system500) to memory504. Processor502may then load the instructions from memory504to an internal register or internal cache. To execute the instructions, processor502may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor502may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor502may then write one or more of those results to memory504. In particular embodiments, processor502executes only instructions in one or more internal registers or internal caches or in memory504(as opposed to storage506or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory504(as opposed to storage506or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor502to memory504. Bus512may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor502and memory504and facilitate accesses to memory504requested by processor502. In particular embodiments, memory504includes random access memory (RAM). This RAM may be volatile memory, where appropriate Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory504may include one or more memories504, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

In particular embodiments, storage506includes mass storage for data or instructions. As an example and not by way of limitation, storage506may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage506may include removable or non-removable (or fixed) media, where appropriate. Storage506may be internal or external to computer system500, where appropriate. In particular embodiments, storage506is non-volatile, solid-state memory. In particular embodiments, storage506includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. This disclosure contemplates mass storage506taking any suitable physical form. Storage506may include one or more storage control units facilitating communication between processor502and storage506, where appropriate. Where appropriate, storage506may include one or more storages506. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

In particular embodiments, I/O interface508includes hardware, software, or both, providing one or more interfaces for communication between computer system500and one or more I/O devices. Computer system500may include one or more of these I/O devices, where appropriate. One or more of these I/O devices may enable communication between a person and computer system500. As an example and not by way of limitation, an I/O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I/O device or a combination of two or more of these. An I/O device may include one or more sensors. This disclosure contemplates any suitable I/O devices and any suitable I/O interfaces508for them. Where appropriate, I/O interface508may include one or more device or software drivers enabling processor502to drive one or more of these I/O devices. I/O interface508may include one or more I/O interfaces508, where appropriate. Although this disclosure describes and illustrates a particular I/O interface, this disclosure contemplates any suitable I/O interface.

In particular embodiments, communication interface510includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between computer system500and one or more other computer systems500or one or more networks. As an example and not by way of limitation, communication interface510may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface510for it. As an example and not by way of limitation, computer system500may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system500may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless network or a combination of two or more of these. Computer system500may include any suitable communication interface510for any of these networks, where appropriate. Communication interface510may include one or more communication interfaces510, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

In particular embodiments, bus512includes hardware, software, or both coupling components of computer system500to each other. As an example and not by way of limitation, bus512may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Bus512may include one or more buses512, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

Herein, a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these, where appropriate. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

Herein, “or” is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A or B” means “A, B, or both,” unless expressly indicated otherwise or indicated otherwise by context. Moreover, “and” is both joint and several, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A and B” means “A and B, jointly or severally,” unless expressly indicated otherwise or indicated otherwise by context.

The scope of this disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the example embodiments described or illustrated herein that a person having ordinary skill in the art would comprehend. The scope of this disclosure is not limited to the example embodiments described or illustrated herein. Moreover, although this disclosure describes and illustrates respective embodiments herein as including particular components, elements, feature, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that a person having ordinary skill in the art would comprehend.