APPARATUS AND METHOD WITH CONTAMINATION DETECTION OF CAMERA LENS

An electronic device and method for detecting contamination of a camera lens, where the electronic device includes at least one camera configured to capture an image, a memory configured to store the image, and a contamination detection model configured to detect a contaminated portion of a lens of the at least one camera, in response to the image being input, and a processor configured to determine whether an operation of the electronic device is hindered by the contaminated portion, in response to the contamination detection model detecting the contaminated portion in the lens of the at least one camera.

CROSS-REFERENCE TO RELATED APPLICATIONS

BACKGROUND

The following description relates to a device and method for detecting contamination of a camera lens.

2. Description of Related Art

Image analysis technology using an artificial intelligence model is being increasingly used for computer vision. In some examples, a deep learning model may be used as an artificial intelligence model for image analysis. The above description has been possessed or acquired by the inventor(s) in the course of conceiving the present disclosure and is not necessarily an art publicly known before the present application is filed.

SUMMARY

In one general aspect, there is provided an electronic device including at least one camera configured to capture an image, a memory configured to store the image, and a contamination detection model configured to detect a contaminated portion of a lens of the at least one camera, in response to the image being input, and a processor configured to determine whether an operation of the electronic device is hindered by the contaminated portion, in response to the contamination detection model detecting the contaminated portion in the lens of the at least one camera.

The contamination detection model may include a model trained to detect a location of the contaminated portion within the image using a grid.

The processor may be configured to determine whether to supplement the contaminated portion with an overlapping area of another image captured by another camera, and to determine whether an operation of the electronic device is hindered by the contaminated portion based on the overlapping area of the another image.

The processor may be configured to update the contamination detection model based on a reference image obtained from the electronic device in an environment of use of the electronic device.

The processor may be configured to update the contamination detection model, using, as training data, the reference image and a label of the reference image determined based on an output of the contamination detection model to which the reference image is input.

The processor may be configured to preprocess the training data and to update the contamination detection model based on the preprocessed training data.

The electronic device may include a communication module configured to communicate with a server, wherein the server may be configured to receive reference images from each of a plurality of electronic devices, to update a super model using each of the reference images, and to update respective contamination detection models stored in each of the plurality of electronic devices using the updated super model, wherein the super model may include weights of all the contamination detection models included in each of the plurality of electronic devices, and wherein each of the respective contamination detection models may include a weight extracted from the super model to be used by the respective electronic device of the plurality of electronic devices.

The server may be configured to extract a weight of the contamination detection model of the electronic device from the super model before updating the contamination detection model of one of the plurality of electronic devices, and to learn the extracted weight using an image received from the electronic device.

In another general aspect, there is provided a method of operating an electronic device, the method including detecting a contaminated portion of a lens of at least one camera based on inputting an image of the at least one camera to a contamination detection model, and determining whether an operation of the electronic device is hindered by the contaminated portion, in response to the contamination detection model detecting the contaminated portion in the lens of the at least one camera.

The contamination detection model may include a model trained to detect a location of the contaminated portion within the image using a grid.

An accuracy of the contamination detection model increases, in response to an increase in a granularity of the grid.

The determining of whether the operation of the electronic device is hindered by the contaminated portion may include determining whether to supplement the contaminated portion with an overlapping area of another image captured by another camera, and determining whether an operation of the electronic device is hindered by the contaminated portion based on the overlapping area of the another image.

The method may include updating the contamination detection model based on a reference image obtained from the electronic device in an environment of use of the electronic device.

The updating of the contamination detection model may include updating the contamination detection model, using, as training data, the reference image and a label of the reference image determined based on an output of the contamination detection model to which the reference image is input.

The updating of the contamination detection model may include preprocessing the training data and updating the contamination detection model based on the preprocessed training data.

The method may include communicating, by the electronic device, with a server through a communication module, receiving, at the server, a reference image from each of a plurality of electronic devices, updating a super model, at the server, using each of the reference images, updating the respective contamination detection models stored in each of the plurality of electronic devices using the updated super model, wherein the super model may include weights of all the contamination detection models included in each of the plurality of electronic devices, and wherein each of the respective contamination detection models may include a weight extracted from the super model to be used by the respective electronic device of the plurality of electronic devices.

The method may include extracting a weight of the contamination detection model from the super model before updating the contamination detection model of the electronic device, and learning the extracted weight using an image received from the electronic device.

In another general aspect, there is provided an electronic device including at least one camera, and a processor configured to load a contamination detection model configured to detect a contaminated portion of a lens of the at least one camera, in response to an input of an image captured by the at least one camera to the contamination detection model, and determine whether a location of the contaminated portion in the lens of the camera hinders an operation of the electronic device, based on the location of the contaminated portion being provided by the contamination detection model using a grid, wherein the contamination detection model includes a convolution layer and is periodically updated for an environment in which the electronic device is used.

The electronic device may be installed in a vehicle, and the processor may be configured to terminate an autonomous driving mode of the vehicle, in response to determining that the contamination portion hinders the operation of the electronic device.

The processor may be configured to activate an output device to notify the user that the autonomous driving mode has terminated and to commence manual driving.

DETAILED DESCRIPTION

Although terms such as “first,” “second,” and “third”, or A, B, (a), (b), and the like may be used herein to describe various members, components, regions, layers, portions, or sections, these members, components, regions, layers, portions, or sections are not to be limited by these terms. Each of these terminologies is not used to define an essence, order, or sequence of corresponding members, components, regions, layers, portions, or sections, for example, but used merely to distinguish the corresponding members, components, regions, layers, portions, or sections from other members, components, regions, layers, portions, or sections. Thus, a first member, component, region, layer, portions, or section referred to in the examples described herein may also be referred to as a second member, component, region, layer, portions, or section without departing from the teachings of the examples.

Throughout the specification, when a component or element is described as being “connected to,” “coupled to,” or “joined to” another component or element, it may be directly “connected to,” “coupled to,” or “joined to” the other component or element, or there may reasonably be one or more other components or elements intervening therebetween. When a component or element is described as being “directly connected to,” “directly coupled to,” or “directly joined to” another component or element, there can be no other elements intervening therebetween. Likewise, expressions, for example, “between” and “immediately between” and “adjacent to” and “immediately adjacent to” may also be construed as described in the foregoing. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. For example, “A and/or B” may be interpreted as “A,” “B,” or “A and B.”

Hereinafter, examples will be described in detail with reference to the accompanying drawings. When describing the examples with reference to the accompanying drawings, like reference numerals refer to like constituent elements and a repeated description related thereto will be omitted.

FIG.1illustrates an example of an electronic device and a server.

The electronic device100may include a camera101, a memory102, a processor103, an output device104, and a communication module105.

In some examples, the electronic device100may include a camera101. In other examples, the electronic device100may include more than one camera101. Any reference to a camera101includes the presence of one camera101or more than one camera101. The camera101may capture an external environment of the electronic device100. The electronic device100may use an image captured by the camera101. For example, when the electronic device100is, or is included in, a vehicle capable of performing autonomous or assisted driving, the vehicle may use images captured by the camera101for autonomous or assisted driving. For example, when the electronic device100is a smartphone which unlocks by recognizing a user's face, the smartphone may recognize the user's face by using images captured by the camera101.

In some examples, the electronic device100may be a device which performs an operation by using an image captured by the camera101. In some examples, the electronic device100may be incorporated in various computing devices such as a mobile phone, a smartphone, a tablet, an electronic-book (e-book) device, a laptop, a personal computer, a desktop, a workstation, or a server, various wearable devices such as a smart watch, smart glasses, or a head-mounted display (HMD), various home appliances such as a smart speaker, a smart TV, or a smart refrigerator, a smart kiosk, an internet of things (IoT) device, a walking assist device (WAD), a drone, or a robot, as devices utilizing an image captured by the at least one camera101.

In another example, the electronic device100may be included in a vehicle. Hereinafter, a vehicle refers to any mode of transportation, delivery, or communication such as, for example, for example, an automobile, a truck, a tractor, a scooter, a motorcycle, a cycle, an amphibious vehicle, a snowmobile, a boat, a public transit vehicle, a bus, a monorail, a train, a tram, an autonomous vehicle, an unmanned aerial vehicle, a bicycle, a drone, and a flying object such as an airplane. In some examples, the vehicle may be, for example, an autonomous vehicle, a smart mobility, an electric vehicle, an intelligent vehicle, an electric vehicle (EV), a plug-in hybrid EV (PHEV), a hybrid EV (HEV), or a hybrid vehicle, an intelligent vehicle equipped with an advanced driver assistance system (ADAS) and/or an autonomous driving (AD) system.

According to some examples, when a lens of the camera101is contaminated, the operation of the electronic device100may be hindered. If the electronic device100operates incorrectly using an image in which contamination is present due to a contaminated lens of the camera101, a problem may occur. For example, when the electronic device100is incorporated in a vehicle capable of performing autonomous driving, the vehicle may cut in between other vehicles, accelerate or decelerate using an image captured by the camera101. If an autonomous vehicle fails to correctly detect an approaching vehicle due to the contaminated lens of the camera101, an accident may occur.

In some examples, the electronic device100may detect a contaminated portion of a camera lens by using an image captured by the camera lens. In some examples, the electronic device100may detect the contaminated portion of the camera lens by using a contamination detection model. The contamination detection model may be a segmentation model trained to detect a contaminated portion of an image and the corresponding lens based on a grid. The contamination detection model may be stored in the memory102and be read by the processor103.

In some examples, the electronic device100may determine whether a contamination that is detected by a contamination detection model is in an area of a lens which may hinder an operation of the electronic device100. The electronic device100may determine whether the contaminated portion detected by the contamination detection model affects the operation of the electronic device100. For example, if the electronic device100is a vehicle capable of autonomous driving, the vehicle may determine whether the contaminated portion has an influence on the autonomous driving by covering over or not detecting a vehicle driving on a lane of a road or on an adjacent road.

In some examples, if it is possible to supplement a loss caused by a contaminated portion of the lens by an overlapping portion of another image, the electronic device100may determine that the contaminated portion does not hinder an operation of the electronic device100. In some examples, the overlapping portion of another image may be an image captured by another camera or a portion that may be sensed by a plurality of sensors (not shown).

In some examples, when a contaminated portion hinders an operation of the electronic device100, the electronic device100may notify a user of the electronic device100. For example, the electronic device100may provide the user with an alarm that an operation is not performable due to a contaminated camera lens, through a display (not shown). In some examples, the alarm may be output through the output device104.

In some examples, when the contaminated portion hinders an operation of the electronic device100, the electronic device100may itself remove the contamination. For example, the electronic device100may remove the contamination by using a wiper and/or washer fluid. In some examples, when a contaminated portion hinders an operation of the electronic device100, the electronic device100may perform an alternative operation of the operation being hindered by the contaminated portion. For example, when the electronic device100is included in a vehicle performing autonomous driving and a contaminated portion hinders the autonomous driving of the vehicle, the vehicle may end the autonomous driving, inform the user that autonomous driving mode has ended, and perform an operation causing the user to drive manually.

At least one camera101may capture a still image or a video. In some examples, the camera101may include one or more lenses, image sensors, image signal processors, or flashes.

The memory102may store various pieces of data, which are used by at least one component of the electronic device100. For example, the memory102may include an image captured by at least one camera101and a contamination detection model for detecting the contamination. In another example, the memory102may store a program (or an application, or software). The stored program may be a set of syntaxes that are coded and executable by the processor103to operate the electronic device100. The memory102may include a volatile memory or a non-volatile memory.

The volatile memory device may be implemented as a dynamic random-access memory (DRAM), a static random-access memory (SRAM), a thyristor RAM (T-RAM), a zero capacitor RAM (Z-RAM), or a twin transistor RAM (TTRAM).

The non-volatile memory device may be implemented as an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic RAM (MRAM), a spin-transfer torque (STT)-MRAM, a conductive bridging RAM (CBRAM), a ferroelectric RAM (FeRAM), a phase change RAM (PRAM), a resistive RAM (RRAM), a nanotube RRAM, a polymer RAM (PoRAM), a nano floating gate Memory (NFGM), a holographic memory, a molecular electronic memory device), or an insulator resistance change memory. Further details regarding the memory102are provided below.

The processor103may control at least one other component of the electronic device100and perform processing of various pieces of data or computations. The processor103may control an overall operation of the electronic device100and may execute corresponding processor-readable instructions for performing operations of the electronic device100. The processor103may execute, for example, software stored in the memory102to control one or more hardware components, such as, camera101of the electronic device100connected to the processor103and may perform various data processing or operations, and control of such components. In some examples, an operation of the electronic device100disclosed in this disclosure may be performed by the processor103. For example, the processor103may detect a contaminated portion of an image captured by at least one camera101, by using a contamination detection model.

The hardware-implemented data processing device103may include, for example, a main processor (e.g., a central processing unit (CPU), a field-programmable gate array (FPGA), or an application processor (AP)) or an auxiliary processor (e.g., a GPU, a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently of, or in conjunction with the main processor. Further details regarding the processor103are provided below.

The communication module105may support wired or wireless communication between the electronic device100and an external electronic device (e.g., the server110). The electronic device100may transmit an image captured by at least one camera101through the communication module105. The electronic device100may communicate with the server110through the communication module105to update a contamination detection model.

In some examples, the processor103may output through the output device104, one or more of an image captured by the camera101, information regarding a contaminated image, information to the user that autonomous driving mode has ended, information commanding the user to commence manual driving. In some examples, the output device104may provide an output to a user through auditory, visual, or tactile channel. The output device104may include, for example, a speaker, a display, a touchscreen, a vibration generator, and other devices that may provide the user with the output. The output device104is not limited to the example described above, and any other output device, such as, for example, computer speaker and eye glass display (EGD) that are operatively connected to the electronic device104may be used without departing from the spirit and scope of the illustrative examples described. In an example, the output device104is a physical structure that includes one or more hardware components that provide the ability to render a user interface, output information and speech, and/or receive user input.

FIG.2illustrates some examples of inference speed of a neural network model.

Referring toFIG.2, an inference speed of a target neural network model trained with high-performance hardware201, can be power intensive, is shown.

A neural network model trained with the high-performance hardware201as a target may include neural networks such as, for example, a CMSIS-NN (common microcontroller software interface standard neural network), a MobileNet-V1, a MobileNet-V2, an Nfs-seg-Lite and a Fovea_NfsNet_LC. The high-performance hardware201may include processing devices such as, for example, a graphics processing unit (GPU), an Exynos Auto V910 (NPU), a Google Pixel 3, an Exynos Auto V910 (Cortex-A CPU), and an Exynos Auto V910. Low-performance hardware202may include processing devices such as, for example, a Cortex-M.

A resource environment of the low-performance hardware202may be a limited environment compared to a resource environment in which the high-performance hardware201learns a neural network model and uses the neural network model.

As illustrated inFIG.2, an inference speed of the low-performance hardware202, when using a neural network model trained with the high-performance hardware201as a target, is significantly less than that of the high-performance hardware201. For example, for Fovea_NfsNet_LC, the inference speed may be 59.64 ms for the high-performance hardware201Exynos Auto V910 (NPU), but the inference speed may be 526,116 ms for the low-performance hardware202Cortex-M.

Therefore, a neural network model may be appropriate, which is used by the low-performance hardware202with a limited resource environment compared to the high-performance hardware201.

In some examples, an electronic device (e.g., the electronic device100ofFIG.1) may be an edge device that directly performs data collection, analysis, and processing. The electronic device, which is an edge device, may have limitations in performance, depending on constraints such as a use environment. Accordingly, the electronic device may include the low-performance hardware202with a relatively limited resource environment compared to the high-performance hardware201. The electronic device may include low-power hardware with a limited resource environment.

A neural network model that performs optimally in an electronic device including the low-performance hardware202with a limited resource environment is described below with reference toFIG.3.

FIG.3illustrates an example of a contamination detection model.

Referring toFIG.3, a contamination detection model300is shown. An electronic device may include the contamination detection model300. An electronic device including low-power hardware with a limited resource environment may detect contaminated portions322and323of a camera lens by using the contamination detection model300.

In some examples, an input image310captured by a camera of an electronic device, may include contaminations311and312caused by corresponding contaminations on a camera lens. Such lens contaminations may include water drops and mud that are deposited on the sensor lens when the camera is exposed to outdoor activities.

In some examples, the electronic device may detect contaminations311and312corresponding to contaminations on a camera lens by using the contamination detection model300. When the input image310is input to the contamination detection model300, an output image320may be output as a result of the input image310being processed by the contamination detection model300.

In some examples, the output image320may include the contaminated portions322and323, and a grid321. The contaminated portions322and323may be partial areas of the grid321corresponding to locations of the contaminations311and312included in the input image310. For example, the first contaminated portion322may be a partial area of a grid corresponding to a location of the first contamination311. The second contaminated portion323may be a partial area of a grid corresponding to a location of the second contamination312.

In some examples, the output image320may display the contaminated portions322and323having different colors on the display104. The colors of the contaminated portions322and323may vary depending on the type of contamination. For example, the output image320may display contaminations from an opaque contaminant, such as mud, in red. The output image320may display transparent contaminations in green. The output image320may have semi-transparent contaminations in blue.

In some examples, the contamination detection model300may be a segmentation model for detecting the contaminated portions322and323caused by contaminations of a camera lens. The contamination detection model300may be a deep learning model detecting the contaminated portions322and323based on the grid321. The contamination detection model300may be a model trained to detect the contaminated portions322and323caused by contaminations on a camera lens from the input image310by using the grid321.

In some examples, the contamination detection model300may be a model that is configured to detect the contaminated portions322and323by a unit in which the contaminated portions322and323may be recognized. In some example, the unit is not a unit of pixels. Referring toFIG.3, in some examples, the contamination detection model300is a model based on an 8×6 grid. In some examples, when a resolution of the grid321, which corresponds to a size of the grid321, increases, the contamination detection model300may detect the contaminated portions322and323more accurately. In other words, when the granularity of the grid321increases, the accuracy of the contamination detection model300may increase. However, when a resolution of the grid321increases, a computation speed may decrease. Accordingly, In some examples, the contamination detection model300may be a model that is trained with a different resolution of a grid321. For example, when an electronic device is an autonomous vehicle driving on a highway, a fast computation speed is needed, so a contamination detection model300that is trained with a relatively low resolution of a grid may be used, compared to resolution of a model that is trained to be used when the vehicle is being driving on a general road. In another example, when an electronic device is an autonomous vehicle and is being parking, accurate detection of a contaminated portion is needed more than a fast computation speed, so the contamination detection model300trained with a relatively high resolution of a grid may be used, compared to a contamination detection model300that is trained for driving on a highway. In some examples, an electronic device may include a plurality of contamination detection models and may change and use the contamination detection model300depending on circumstances.

In some examples, the contamination detection model300may include one convolution layer. In some examples, the contamination detection model300may not include an upsampling layer and/or a deconvolution layer.

Hereinafter, training and updating the contamination detection model300is further described.

FIGS.4A and4Billustrate examples of learning and updating a contamination detection model.

FIG.4Ais a diagram illustrating training of a contamination detection model300.FIG.4Bis a diagram illustrating updating for a personalization of the contamination detection model300.

In some examples, the contamination detection model300may be a model trained by first training data410. The first training data410may include an image411and ground truth412. The image411may be an image including contamination. The ground truth412may be an image in which contamination and an uncontaminated area are marked from the image411. The first training data410may be training data in which the ground truth412is labeled on the image411.

In some examples, training of the contamination detection model300using the first training data410may be performed in a server (e.g., the server550ofFIG.5or the server110ofFIG.1). The contamination detection model300trained in a server may be applied to an electronic device, such as the electronic device100.

In some examples, the contamination detection model300may output a result image422when a reference image421is input. In some examples, the reference image421is an image captured by a camera included in an electronic device, as an input. In some examples, a resolution of the reference image421may be 1280×720. In an example, since the contamination detection model300may include one convolution layer, a resolution of the result image422may be lower than that of the reference image421. For example, the resolution of the result image422may be 44×22.

The contamination detection model300trained using the first training data410may be a universal model that is usable in all electronic devices described above. However, an environment in which an electronic device including the contamination detection model300is used may differ depending on the use of the electronic device. Accordingly, the contamination detection model300may be updated to better detect a contaminated portion of a camera lens in the environment in which the electronic device is being used.

The reference image421newly collected by an electronic device using a camera is generally not labeled with a ground truth. Therefore, to update the contamination detection model300with the reference image421, the result image422, which is output by inputting the reference image421to the contamination detection model300may be used.

In some examples, a result image422, which has low resolution compared to the reference image421, may be upsampled. The result image422may be upsampled to have the same resolution as the reference image421. An upsampling image423may have the same resolution as the reference image421.

In some examples, the contamination detection model300may be updated by using second training data424including a label in which the reference image421and the upsampling image423form a labeled pair. The upsampling image423of the second training data424may serve a similar purpose as the ground truth412of the first training data410.

In some examples, the second training data424may be preprocessed. A preprocessed image may be an image generated by resizing and cropping the second training data424. The electronic device may generate various pieces of training data by preprocessing the second training data424.

In some examples, updating of the contamination detection model300by using the second training data424may be performed by a processor of an electronic device or a server (e.g., the server110ofFIG.1or the server550ofFIG.5) communicating with an electronic device by wire and/or wirelessly. A processor or a server of an electronic device may update the contamination detection model300when the electronic device is in a standby mode.

By updating the contamination detection model300using the reference image421captured by an electronic device, the contamination detection model300may be updated to detect a contaminated portion that is suitable to an environment in which the electronic device is used. An accuracy of detecting the contaminated portion of the contamination detection model300, which is updated to detect the contaminated portion suitably to the environment in which the electronic device is used, may increase.

Hereinafter, training of a super model, performed in a server, is further described.

FIG.5illustrates an example of a server training a super model. In addition to the description ofFIG.5below, the descriptions ofFIGS.1-4are also applicable toFIG.5.

Referring toFIG.5, a first electronic device510, a second electronic device520, a third electronic device530, and a server550are shown.

The first electronic device510, the second electronic device520, and the third electronic device530may be instances of the electronic device100ofFIG.1. The server550may be an instance of the server110forFIG.1. The first electronic device510may store a first contamination detection model511. The second electronic device520may store a second contamination detection model521. The third electronic device530may store a third contamination detection model531. The first contamination detection model511, the second contamination detection model521, and the third contamination detection model531may be examples of the contamination detection model300ofFIGS.3,4A, and4B.

Contamination detection models stored in the first electronic device510, the second electronic device520, and the third electronic device530may be different models. For example, the first contamination detection model511, the second contamination detection model521, and the third contamination detection model531may have different weights for detecting a contaminated portion.

The server550may communicate with the first electronic device510, the second electronic device520, and the third electronic device530by wire and/or wirelessly. The server550may include a super model500. The super model500may be a model including all the weights of the first contamination detection model511, second contamination detection model521, and third contamination detection model531stored in the first electronic device510, second electronic device520, and third electronic device530, respectively. The first contamination detection model511, second contamination detection model521, and third contamination detection model531stored in the first electronic device510, second electronic device520, and third electronic device530, respectively, may be models in which only weights that are needed for a corresponding electronic device are extracted from among weights included in the super model500and such extracted weights are provided to the respective electronic devices510-530for use as the individual contamination detection modules511-531. That is, the first contamination detection model511, second contamination detection model521, and third contamination detection model531stored in the first electronic device510, second electronic device520, and third electronic device530, respectively, may be submodels extracted from the super model500and provided to the electronic devices510-530.

In some examples, the server550may periodically receive images captured by the first electronic device510, second electronic device520, and third electronic device530. For example, the server550may periodically receive a first reference image512, a second reference image522, and a third reference image532from the first electronic device510, second electronic device520, and third electronic device530respectively.

In some examples, the server550may update the super model500using reference images received from the first electronic device510, second electronic device520, and third electronic device530, respectively. The server550may train the super model500offline by using reference images received from the first electronic device510, second electronic device520, and third electronic device530. A weight included in the super model500may be changed due to the update.

In some examples, the server550may update the first contamination detection model511, second contamination detection model521, and third contamination detection model531stored in the first electronic device510, second electronic device520, and third electronic device530, respectively, using an updated super model500. The server550may update weights of the first contamination detection model511, second contamination detection model521, and third contamination detection model531stored in the first electronic device510, second electronic device520, and third electronic device530, respectively, using corresponding updated weights of the updated super model500.

By updating a weight of the super model500included in the server550, an accuracy of a contamination detection model using the weight may generally increase.

An updating of the super model500using the first reference image512, second reference image522, and third reference image532received from the first electronic device510, second electronic device520, and third electronic device530, respectively, may be a universal update applicable to all electronic devices. Therefore, before updating the contamination detection model stored in the electronic device by using the super model500, a fine tuning may be needed to extract and personalize a weight included in the corresponding contamination detection sub-model. Hereinafter, a fine tuning is described.

FIG.6illustrates an example of a server650performing a fine tuning. In addition to the description ofFIG.6below, the descriptions ofFIGS.1-5are also applicable toFIG.6.

The server650may update the first contamination detection model611stored in the first electronic device610using the updated super model600. The first electronic device610may be an instances of the electronic device100ofFIG.1, and the server650may be an instance of the server110forFIG.1or the server550ofFIG.5. A server may fine tune the first contamination detection model611to better detect contamination in an environment in which the first electronic device610is used.

In some examples, the server650may update a contamination detection model of any one of a plurality of electronic devices communicating with the server650by wire and/or wirelessly. For example, the server650may update the first contamination detection model611of the first electronic device610by sending updated weights from the server650by wire and/or wirelessly.

In some examples, the server650may extract a weight included in a contamination detection model to be updated, from the super model600. For example, the server650may extract weights W1, W3, W5, W7, and W9 included in the first contamination detection model611(for example, based on an identity or trait of the first electronic device510), which is a contamination detection model to be updated, from the super model600. A weight extracted by the server650may be an updated weight, as a weight that has been learned (updated) from a plurality of reference images received from a plurality of electronic devices.

In some examples, the server650may learn an extracted weight using third training data605. The server650may fine tune the extracted weight by using the third training data605. The server650may update the extracted weight by using the third training data605. The third training data605may be training data in which a reference image601received from an electronic device which will update a contamination detection model and ground truth602of the reference image601are labeled. For example, the reference image601received from an electronic device which will update a contamination detection model may be the first reference image512received from the first electronic device510ofFIG.5and may be used as a training sample as described with reference toFIG.5.

In some examples, the server650may update a contamination detection model of an electronic device using a weight learned by the third training data605. For example, the server650may update the first contamination detection model611using a weight learned by the third training data605.

The server650may better detect a contaminated portion of a lens in an environment in which an electronic device is used, by using the method of fine tuning a weight described above. The server650may personalize a contamination detection model by using the method of fine tuning a weight described above, to better detect a contaminated portion according to a user of an electronic device.

FIG.7illustrates an example of a method of detecting a contaminated portion of a lens of a camera. The operations ofFIG.7may be performed in the sequence and manner as shown. However, the order of some operations may be changed, or some of the operations may be omitted, without departing from the spirit and scope of the shown example. Additionally, operations illustrated inFIG.7may be performed in parallel or simultaneously. One or more blocks ofFIG.7, and combinations of the blocks, can be implemented by special purpose hardware-based computer that perform the specified functions, or combinations of special purpose hardware and instructions, e.g., computer or processor instructions. For example, operations710through740may be performed by a computing apparatus (e.g., processor103or the server110ofFIG.1). In addition to the description ofFIG.7below, the descriptions ofFIGS.1-6are also applicable toFIG.7and are incorporated herein.

Referring toFIG.7, in operation710, the electronic device100may detect a contaminated portion of a lens of a camera based on inputting an image to a contamination detection model.

In operation720, the electronic device100may determine whether an operation of an electronic device is hindered by the contaminated portion.

In operation730, the electronic device100may determine whether to supplement the contaminated portion with an overlapping area of another image captured by another camera.

In operation740, the electronic device100may update the contamination detection model based on a reference image obtained from the electronic device in an environment of use of the electronic device.

FIG.8illustrates an example of a method of training one or more models. In some examples, the method of training the models may be performed at a server. The operations ofFIG.8may be performed in the sequence and manner as shown. However, the order of some operations may be changed, or some of the operations may be omitted, without departing from the spirit and scope of the shown example. Additionally, operations illustrated inFIG.8may be performed in parallel or simultaneously. One or more blocks ofFIG.8, and combinations of the blocks, can be implemented by special purpose hardware-based computer that perform the specified functions, or combinations of special purpose hardware and instructions, e.g., computer or processor instructions. For example, operations810through830may be performed by a computing apparatus (e.g., the server550ofFIG.5or the server110ofFIG.1). In addition to the description ofFIG.8below, the descriptions ofFIGS.1-7are also applicable toFIG.8and are incorporated herein.

Referring toFIG.8, in operation810, the server, for example server550ofFIG.5, may receive a reference image from each of a plurality of electronic devices.

In operation820, the server, for example server550ofFIG.5, may update a super model stored at the server, for examples super model550or super model600, using each of the reference images.

In operation830, the server, for example server550ofFIG.5, may update the respective contamination detection models stored in each of the plurality of electronic devices using the updated super model.

FIG.9illustrates an example of an accuracy of a contamination detection model according to an update and a fine tuning of the contamination detection model.

Referring toFIG.9, a table900showing accuracy of submodels 1 to 4 in Experiments 1 to 4.

The submodels 1 to 4 may be models in which only some of the weights are extracted from among weights included in a super model (e.g., the super model500ofFIG.5).

Experiment 1 illustrates an example of an accuracy of a submodel when the submodel is simply extracted from a super model. Experiment 2 illustrates an example of an accuracy of a of a submodel when the update described with reference toFIGS.4A and4Bis additionally performed on the submodel extracted in Experiment 1. Experiment 3 illustrates an example of an accuracy of a submodel extracted from a super model trained by the method described with reference toFIG.5. Experiment 4 illustrates an example of an accuracy of a submodel when the fine tuning described with reference toFIG.6is additionally performed on the submodel extracted in Experiment 3.

Comparing Experiments 1 and 2, it may be confirmed that the accuracy increases by performing the update described with reference toFIGS.4A and4B. Comparing Experiments 2 and 3, it may be confirmed that the accuracy of the submodel extracted from the super model trained by the method described with reference toFIG.5is higher than that of the submodel on which only the update of Experiment 2 is performed. Comparing Experiments 3 and 4, it may be confirmed that an accuracy of a submodel may further increase if the fine tuning described with reference toFIG.6is additionally performed on the trained super model. Comparing Experiments 2 and 4, it may be confirmed that an accuracy of a submodel personalized through super model training and fine tuning may be higher than that of a submodel personalized through the update described with reference toFIGS.4A and4B.