Patent Description:
A dishwasher may be defined as a device used to remove food and oil stains on a surface of a dish, wherein the dish to be washed is placed therein. The food and the oil stains can be removed by spraying washing water with detergent. For example, the washing water can be sprayed into an inside of a tub at high pressure and the sprayed washing water can contact the dishes to wash the food and oil stains on the surface of the dishes. However, as the stain can include hard or sharp materials such as bones and the washing water can be sprayed at high pressure, the objects in the stain moving can move upward by the spraying of the washing water and collide with an inner surface of the tub, thereby scratching the inner surface of the tub.

<CIT>) relates to a dishwasher, and a method and computer program for operating a dishwasher.

<CIT>) relates to a method of automatically assessing the displacement of an object in a washing area of a dishwashing machine.

The present application describes a dishwasher that can detect a failure of a dish recognition camera disposed inside the dishwasher by image processing a dish image and quickly respond to the failure of the dish recognition camera.

The present application also describes a dishwasher that can determine various types of failures by detecting an abnormal operation of a dish recognition camera in various ways and respond to the failures differently according to the types of failures.

The present application further describes a dishwasher that can prevent malfunction of the dishwasher by recognizing the failure of the dish recognition camera in advance.

These objects are achieved with the features of the independent claims. Preferred aspects are defined in the dependent claims.

Hereinafter, one or more implementations of the present disclosure are described in detail with reference to the accompanying drawings. Same reference numerals may be used to refer to same or similar components.

An inner structure of the dishwasher is described in detail with reference to <FIG>. <FIG> is a perspective view of a dishwasher with a door being closed. <FIG> is a perspective view of a dishwasher with a door being opened.

Referring to <FIG>, an exterior of the dishwasher includes a case <NUM> defining an opening at a front side thereof and a door <NUM> to close the opening at the front side of the case. The case <NUM> and the door <NUM> disposed in front of the dishwasher can close an inner space thereof to prevent or otherwise restrict washing water or detergent from leaking to the outside while washing the dishes. Referring to <FIG>, when a door <NUM> is opened, the door <NUM> can form a right angle of <NUM> degrees with a case <NUM>. Inner components of a dishwasher can include a tub <NUM> to receive washing water, a sump <NUM> disposed at a bottom surface of the tub to collect and spray the washing water by filtering out foreign substances, a nozzle <NUM> connected to and rotating above the sump, and configured to spray the washing water into the tub, and an upper rack <NUM> and a lower rack <NUM> disposed at an upper portion and a lower portion thereof in the tub, respectively, and configured to accommodate a plurality of dishes.

The dishwasher includes a dish recognition camera in an inner surface of the tub to recognize a dish placed in the dishwasher. <FIG> is a side cross-sectional view of an example dishwasher including dish recognition cameras.

Referring to the drawings, outer components of the dishwasher <NUM> can include a case <NUM> defining an opening at a front side thereof and a door <NUM> to close the opening at the front side of the case <NUM>, wherein the case <NUM> and the door <NUM> can define an outer appearance of the dishwasher <NUM>. A tub <NUM> can be disposed in the dishwasher to receive the washing water and a sump <NUM> can be disposed on a bottom surface of the tub to collect the washing water and spray the washing water by filtering out foreign substances.

For example, a nozzle <NUM> can be disposed in the tub <NUM> and sprays the washing water toward the upper rack <NUM> and the lower rack <NUM>. In addition, a washing water flow path <NUM> can be defined on an inner side surface of the tub <NUM> to supply the washing water to an upper nozzle 14a and a top nozzle 14c.

In addition, according to an implementation of the present disclosure, a dish recognition camera <NUM> can be disposed inside the tub <NUM>. Referring to <FIG>, a plurality of dish recognition cameras <NUM> can be disposed inside the tub <NUM>. For example, dish recognition cameras 20a and 20b can be disposed above the upper rack <NUM> to capture the dish accommodated in the upper rack <NUM> and dish recognition cameras 20c and 20d can be disposed at a middle portion between the lower rack <NUM> and the upper rack <NUM> to capture the dishes accommodated in the lower rack <NUM>.

In some implementations, the dish recognition camera <NUM> can be disposed at a side surface of the tub <NUM> or the door <NUM>. For example, as shown in <FIG>, the tub <NUM> or the door <NUM> may define a groove in the inner surface thereof and the dish recognition camera <NUM> may be inserted into the groove. In some implementations, the groove can be formed using a panel. The panel can be made of transparent material such as glass or plastic.

In some implementations, the dish recognition camera can be disposed inside a groove covered by the panel and formed inside the tub. Nevertheless, the failure rate of the dish recognition camera during the operation of the dishwasher is high. As the inside of the tub of the dishwasher is in a high temperature and high humidity environment, an image sensor of the dish recognition camera can be damaged. In some implementations, a lens of the dish recognition camera can be damaged or the panel can be scratched due to physical collision with the dishes, the food and oil stains, or the washing water. Furthermore, various types of failures can occur due to the food and oil stains and the washing water on the surface of the lens and foreign substances deposited on the surface of the lens.

In some implementations, when the failure of the dish recognition camera is not known in advance, the dishwasher obtains an incorrect dish image from the dish recognition camera due to the failure of the dish recognition camera, thereby resulting in malfunction of the dishwasher.

<FIG> is a schematic block diagram showing example components of an example dishwasher. Referring to the drawing, a dish recognition camera <NUM> can be disposed on an inner surface of a tub or a door and can generate a dish image by capturing an upper portion of a rack. In some implementations, the dish recognition camera <NUM> can include a converting device to convert an external optical image signal into an electric image signal using an image sensor. For example, an image sensor can include a CMOS(Complementary Metal-Oxide Semiconductor) image sensor, i.e., CIS or a CCD(Charge-Coupled Device) sensor.

A lighting device <NUM> can be disposed at a side surface or an upper surface of the tub to illuminate an upper portion of an upper rack <NUM> or a lower rack <NUM>. In some cases, a plurality of lighting devices <NUM> can be provided and controlled by a controller <NUM>. The controller <NUM> can include a device controller <NUM>, an image receiver <NUM>, an image processor <NUM>, an abnormal operation determiner <NUM>, and an abnormal operation processor <NUM>.

The device controller <NUM> can control the dish recognition camera <NUM> and the lighting device <NUM>. In some cases, power can be applied to the lighting device <NUM>, and the dish recognition camera <NUM> can capture the upper portion of the rack to generate a dish image based on a control signal from the device controller <NUM>. The lighting device <NUM> and the dish recognition camera <NUM> may be sequentially or simultaneously controlled based on the control signal from the device controller <NUM>, and the control signal may control on/off of the lighting device or control illuminance.

In some implementations, the device controller <NUM> can control operation steps of the dishwasher. For example, the device controller <NUM> may sequentially perform preliminary cleaning, main cleaning, rinsing, heat rinsing, and drying.

The image receiver <NUM> can receive the dish image and a reference image captured by the dish recognition camera <NUM>. Subsequently, the image receiver <NUM> can transmit, to the image processor <NUM>, the received dish image and reference image according to a request from the image processor <NUM>.

The image processor <NUM> can process the received dish image using various methods. The image processing may be performed, by comparing a number of static pixels of the dish image, comparing brightness values of the pixels of the dish images, or comparing with the reference image.

An abnormal operation determiner <NUM> can determine whether the dish recognition camera <NUM> is abnormally operated based on a result of processing the image by the image processor <NUM>. The dish recognition camera <NUM> can perform various abnormal operations inside the tub. For example, the image sensor of the dish recognition camera may be damaged due to high temperature and high humidity inside the tub of the dishwasher, and the lens or the panel may be damaged due to physical collision with the dishes, the food or oil stains, or the washing water. In some implementations, the food and oil stains and the washing water may adhere to the surface of the lens and the foreign substances may be deposited on the surface of the lens. The abnormal operation determiner <NUM> can distinguish and determine the various abnormal operations based on the result of processing the image.

The abnormal operation processor <NUM> can respond to the abnormal operations according to the different types of abnormal operations. For example, the abnormal operation processor <NUM> can display a message notifying the determined abnormal operation on a display of the dishwasher or transmit abnormal operation information to a server to request a response thereto.

<FIG> is a flowchart showing an example operation of image processing and determining an abnormal operation to detect panel contamination or lens damage. <FIG> show examples of image processing to detect panel contamination or lens damage. Hereinafter, in an implementation of the present disclosure, a method for determining an abnormal operation is described with reference to <FIG>.

A panel contamination can include contamination due to water spots generated by the washing water and deposits of foreign substances. A lens damage can include damage due to collision of the lens with the stain. As the panel contamination or lens damage may occur on the surface of the lens or the panel, the panel contamination or lens damage can have a fixed position and shape in the dish image regardless of a change in capturing angle of the dish recognition camera <NUM> or a change in the surrounding environment of the dish. Therefore, the panel contamination or the lens damage may be detected by comparing a plurality of dish images and detecting static pixels with no movement or no color change for a long period of time.

After the image receiver <NUM> obtains a first dish image and a second dish image captured at different time points or different angles by the dish recognition camera <NUM> (<NUM>), the image processor <NUM> compares the received first dish image and second dish image (<NUM>). The two dish images <NUM> shown in <FIG> are examples of a first dish image 601a and a second dish image 601b.

The image processor <NUM> compares the first dish image with the second dish image and extract static pixels based on the first dish image and the second dish image (<NUM>).

In some implementations, the image processor <NUM> can detect a change in color by image processing based on the dish images (<NUM>). In some implementations, a dish image can include a plurality of pixels, and each pixel has a color value. The image processor <NUM> may use a predetermined color model to classify the static pixel based on the color value. For example, the color model may include one of RGB (Red, Green, and Blue), CMYK (Cyan, Magenta, Yellow, and Black), HSV (Hue, Saturation, and Value), HSL (Hue, Saturation, or Lightness), or HEX (Hexadecimal color code). However, the present disclosure is not limited thereto and can be used with other color models used to express colors using a digital method.

The image processor <NUM> can classify a pixel from the dish images as a static pixel when there is no change in color value, then detect a plurality of adjacent static pixels among the classified static pixels, and classify an area of the plurality of adjacent static pixels as a contamination area (<NUM>). <FIG> shows an example static pixel image <NUM>, which is a plurality of pixels with no change in color value among pixels of each of a first dish image 601a and a second dish image 601b. <FIG> shows an example contamination area <NUM> classified by detecting a plurality of adjacent static pixels based on the static pixel image <NUM>.

An abnormal operation determiner <NUM> presets a number of reference pixels for determining an abnormal operation. The abnormal operation determiner <NUM> determines contamination of the panel or damage of the lens of the dish recognition camera <NUM> based on the number of static pixels in the classified contamination area <NUM> exceeding a preset number of reference pixels.

In some implementations, the image processor <NUM> can perform image processing and detect a change in an edge based on the dish images (<NUM>). For example, the image processor <NUM> can generate a boundary according to a difference in color values of the pixels from a dish image, thereby extracting an edge. <FIG> shows an example first edge image <NUM> extracted from the first dish image 601a.

In some implementations, the image processor <NUM> can compare the first edge image <NUM> extracted from the first dish image and a second edge image extracted from the second dish image (<NUM>) and detect a change in edge (<NUM>). In some implementations, the image processor <NUM> can classify a pixel as a static pixel when there is no edge change (<NUM>), and then classify an area of a plurality of adjacent static pixels among the classified static pixels as a contamination area (<NUM>).

In some cases, the abnormal operation determiner <NUM> can preset a number of reference pixels for determining an abnormal operation. Based on the number of static pixels in the classified contaminated area exceeding the preset number of reference pixels, the abnormal operation determiner <NUM> may determine that panel contamination or lens damage of the dish recognition camera <NUM> has occurred (<NUM>).

Meanwhile, in an environment in which washing water is sprayed, it can be difficult to accurately classify lens damage or panel contamination using static pixels. Accordingly, the device controller <NUM> may preset operation timing in order for the dish recognition camera <NUM> to capture the first dish image and the second dish image in a process in which the washing water is not sprayed.

In some implementations, an image receiver <NUM> can obtain a first dish image after a non-washing step of a first cycle in which washing water is not sprayed and obtain a second dish image after a non-washing step of a second cycle in which washing water is not sprayed.

For example, the dishwasher can include a plurality of operation steps including preliminary washing, main washing, rinsing, heat rinsing, and drying in one cycle and the cycle can be repeated. As a step in which the washing water is not sprayed can be a drying step, a non-washing step may be the drying step. The preset operation time point at which the dish recognition camera <NUM> generates a dish image can be a time point at which the drying step ends. The dish recognition camera <NUM> may generate a first dish image after the drying step of the first cycle ends and generate a second dish image after the drying step of the second cycle ends.

Images are referred to as the first dish image and the second dish image, the image processor compares a plurality of dish images, extracts a static pixel and may determine a contamination area based on a plurality of three or more dish images. The abnormal operation determiner <NUM> can accurately determine panel contamination or lens damage by classifying the contamination area based on the three or more dish images.

An abnormal operation processor <NUM> can display a panel contamination or lens damage message on a display of the dishwasher <NUM> and transmit an error code through wireless communication with a server. Therefore, a user may directly wash the panel or the lens by recognizing the panel contamination or the lens damage, or a repair technician can be requested by the server and replace the panel or the lens, thereby reducing a possibility of malfunction of the dishwasher due to the damage to the image sensor.

<FIG> is a flowchart of an example operation of image processing determining an abnormal operation to detect a defective pixel. <FIG> show an example dish image according to an image processing method to detect a defective pixel. Hereinafter, a method for determining an abnormal operation is described with reference to <FIG>.

A defective pixel can be caused by damage to an image sensor disposed inside a dish recognition camera <NUM>. The defective pixel may have no change in brightness value even if there is a change in brightness of light transmitted to the dish, which is a light capturing object. Therefore, by detecting a pixel with a constant brightness value, a defective pixel can be detected in an environment in which the brightness of light illuminating the dishes is rapidly changed by opening and closing a door and controlling on/off of a light.

<FIG> shows an example first dish image <NUM> generated when a door is open. <FIG> shows an example second dish image <NUM> generated when a door is closed. Referring to <FIG>, when the user opens the door <NUM> of the dishwasher <NUM> (<NUM>), an image receiver <NUM> receives the first dish image generated from a dish recognition camera <NUM> (<NUM>). When the user closes the door <NUM> of the dishwasher <NUM> (<NUM>), the image receiver <NUM> receives the second dish image generated from the dish recognition camera <NUM> (<NUM>). In some cases, as a sudden change in brightness of light may need to occur, the door <NUM> can be quickly opened and closed by the user.

An image processor <NUM> compares brightness values of pixels at the same position based on the obtained first dish image and second dish image (<NUM>) and then detect a pixel with no change in brightness value (<NUM>). Referring to <FIG> and <FIG>, a pixel <NUM> inside a circle indicated by dotted lines in the first dish image in <FIG> and a pixel <NUM> inside a circle indicated by dotted lines in the second dish image in <FIG> have the same brightness value. Therefore, the image processor <NUM> can detect the pixels <NUM> and <NUM> with no change in brightness value.

In some implementations, if the surrounding environment is dark at a time of opening and closing the door, the brightness values of all pixels may not be changed regardless of opening and closing of the door. Accordingly, by providing a rapid change in brightness of light inside the tub of the dishwasher through lighting control in a state in which the door is closed, the difference in brightness values may be clearly detected.

In some implementations, the image processor <NUM> may compare the dish images before and after the door is opened and closed and also compare additional dish images obtained by controlling on/off of light when the door is closed.

Referring to <FIG>, after obtaining the second dish image, the device controller <NUM> can apply power to the lighting device <NUM> (<NUM>) and then the image receiver <NUM> may obtain a third dish image generated by the dish recognition camera <NUM> (<NUM>). The image processor <NUM> may compare brightness values of pixels at the same position based on the obtained first dish image to the third dish image (<NUM>) and detect the pixel with no change in brightness value (<NUM>). Subsequently, the abnormal operation determiner <NUM> determines the detected pixel as a defective pixel (<NUM>).

Referring to <FIG>, the pixel <NUM> inside the circle indicated by the dotted lines in the first dish image in <FIG>, the pixel <NUM> inside the circle indicated by the dotted lines in the second dish image in <FIG>, and a pixel <NUM> inside a circle indicated by dotted lines in a third dish image in <FIG> can have the same brightness value. As shown, as the brightness values of the above pixels are not changed while brightness values of other surrounding pixels in the first dish image to the third dish image are changed, the image processor <NUM> may detect the pixels <NUM>, <NUM>, and <NUM> with no change in brightness value.

The abnormal operation determiner <NUM> may classify a pixel as a defective pixel when the pixel has the same detected brightness value and determine that the image sensor of the dish recognition camera <NUM> is damaged. The abnormal operation processor <NUM> may display an image sensor error message on the display of the dishwasher <NUM> and transmit an error code of the image sensor through wireless communication with the server. Therefore, the user recognizes the damage to the image sensor and replaces the image sensor directly, or a repair technician is requested by the server and replaces the image sensor, thereby reducing a possibility of malfunction of the dishwasher due to the damage to the image sensor.

<FIG> is a flowchart of an example operation of image processing and determining an abnormal operation to detect a defective color filter. <FIG> and <FIG> show an example dish image obtained when a color filter is normal, and an example dish image obtained when a color filter is defective, respectively. Hereinafter, in an implementation of the present disclosure, the method for determining the abnormal operation is described with reference to <FIG>.

A color filter may be defective when a specific color component is omitted from a color value of a pixel in a dish image. <FIG> shows an example reference image <NUM> obtained when a color filter is normal. <FIG> shows an example color filter damage image <NUM> obtained when a color filter is in a defective state, that is, a specific color component is omitted from the color value of the dish image. As shown, if the specific color component is omitted, the corresponding color component may not display in the dish image as if a color filter is used. Therefore, the reference image <NUM> may be needed to determine whether there is an abnormality in color component or not.

Specifically, an identifier 858a may be disposed at one side of the upper rack <NUM> or the lower rack <NUM>. The identifier is an identification element, which is a standard to determine normality or abnormality of color component, wherein two or more identifiers may be disposed in the rack, and may have different color values. In some implementations, the identifier may include an identifier 858a disposed in the rack as shown in <FIG> and may also include identifiers 858b, 858c, 858d, and 858e disposed at an upper portion of the rack as shown in <FIG>. The rack of the present disclosure can include a plurality of identifiers to accurately determine failure or non-failure of the color filter of a dish recognition camera.

The dish recognition camera <NUM> can generate a dish image by capturing the upper portion of the rack including the identifier, and the image processor <NUM> may preset a dish image generated by the dish recognition camera in a normal state as the reference image <NUM>. Referring to <FIG>, when power is applied to a lighting device <NUM> (<NUM>), a controller <NUM> may acquire a dish image generated by the dish recognition camera <NUM> (<NUM>). The dish image obtained in an implementation of the present disclosure is described below on the assumption that it is the color filter damaged image <NUM> in <FIG>.

In some implementations, the image processor <NUM> can extract a color value of a pixel of a portion corresponding to the identifier 858a of the color filter damage image <NUM> (<NUM>). Subsequently, the image processor <NUM> compares the color value of the pixel at the portion corresponding to the identifier of the dish image with that of the pixel at the portion corresponding to the identifier of a preset reference image (<NUM>). A color value of a pixel at a different portion than the portion corresponding to the identifier can be changed based on material and color of the dish. Therefore, the image processor <NUM> can compare the color value of the pixel with the color value of a pixel at the portion corresponding to the identifier having a fixed color value.

The abnormal operation determiner <NUM> determines a state of the color filter of the dish recognition camera <NUM> as a failure state based on the color value of the pixel at the portion corresponding to the identifier of the dish image being different from that of the pixel at the portion corresponding to the identifier of the reference image (<NUM>).

The abnormal operation processor <NUM> may display a color filter error message on the display of the dishwasher <NUM> and transmit a color filter error code through wireless communication with the server. Therefore, the user recognizes damage to the color filter of the dish recognition camera <NUM> and directly replaces the dish recognition camera <NUM>, or a repair technician can be requested by the server and replaces the color filter, thereby reducing malfunction of the dishwasher due to the damage to the color filter.

The dishwasher according to an implementation of the present disclosure may self-detect a failure of the dish recognition camera disposed inside the dishwasher by image processing based on a dish image and respond to the failure of the dish recognition camera.

Claim 1:
A dishwasher, comprising:
a case (<NUM>) defining an outer appearance and an opening at a front side thereof;
a door (<NUM>) configured to close the open front side of the case (<NUM>);
a tub (<NUM>) configured to provide a washing chamber;
a rack (<NUM>, <NUM>) disposed inside the tub (<NUM>) and configured to accommodate a dish;
a dish recognition camera (<NUM>, 20a-20d) disposed on at least one of an inner surface of the door (<NUM>) or an inner surface of the tub (<NUM>) and configured to capture an upper portion of the rack (<NUM>, <NUM>); and
a controller (<NUM>) configured to control driving of the dishwasher and the dish recognition camera (<NUM>, 20a-20d),
wherein the controller (<NUM>) is configured to:
obtain a first dish image (601a) and a second dish image (601b) captured at different time points by the dish recognition camera (<NUM>, 20a-20d) based on a preset operation,
characterized in that
the controller is further configured to:
process the obtained first dish image (601a) and second dish image (601b), and
determine an abnormal operation of the dish recognition camera (<NUM>, 20a-20d) based on a result of processing the images,
extract static pixels from pixels of each of the first dish image (601a) and the second dish image (601b);
classify a contamination area based on the static pixel and determine a state of the dish recognition camera (<NUM>, 20a-20d) as a lens damage state or a panel contamination state based on a number of static pixels in the contamination area being exceeding a number of preset reference pixels,
or
based on a brightness value of each of pixels of the first dish image (601a) being identical to a brightness value of each of pixels of the second dish image (601b) by comparing the brightness value of each of pixels of the first dish image (601a) with the brightness value of each of pixels of the second dish image (601b), determine that an image sensor of the dish recognition camera (<NUM>, 20a-20d) is damaged,
or
determine a state of a color filter of the dish recognition camera (<NUM>, 20a-20d) as a failure state based on a color value of a pixel at a portion corresponding to an identifier of the first dish image (601a) or the second dish image (601b) being different from a color value of a pixel at a portion corresponding to an identifier of a preset reference image.