Patent ID: 12251065

Repeat use of reference characters in the present specification and drawings is intended to represent the same or analogous features or elements of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Reference now will be made in detail to embodiments of the invention, one or more examples of which are illustrated in the drawings. Each example is provided by way of explanation of the invention, not limitation of the invention. In fact, it will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the scope or spirit of the invention. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present invention covers such modifications and variations as come within the scope of the appended claims and their equivalents.

As used herein, the terms “includes” and “including” are intended to be inclusive in a manner similar to the term “comprising.” Similarly, the term “or” is generally intended to be inclusive (i.e., “A or B” is intended to mean “A or B or both”). Approximating language, as used herein throughout the specification and claims, is applied to modify any quantitative representation that could permissibly vary without resulting in a change in the basic function to which it is related. Accordingly, a value modified by a term or terms, such as “about,” “approximately,” and “substantially,” are not to be limited to the precise value specified. In at least some instances, the approximating language may correspond to the precision of an instrument for measuring the value. For example, the approximating language may refer to being within a 10 percent margin.

As used herein, the term “article” may refer to, but need not be limited to dishes, pots, pans, silverware, and other cooking utensils and items that can be cleaned in a dishwashing appliance. The term “wash cycle” is intended to refer to one or more periods of time during which a dishwashing appliance operates while containing the articles to be washed and uses a detergent and water, preferably with agitation, to e.g., remove soil particles including food and other undesirable elements from the articles. The term “rinse cycle” is intended to refer to one or more periods of time during which the dishwashing appliance operates to remove residual soil, detergents, and other undesirable elements that were retained by the articles after completion of the wash cycle. The term “drain cycle” is intended to refer to one or more periods of time during which the dishwashing appliance operates to discharge soiled water from the dishwashing appliance. The term “cleaning cycle” is intended to refer to one or more periods of time that may include a wash cycle, rinse cycle, and/or a drain cycle. The term “wash fluid” refers to a liquid used for washing and/or rinsing the articles and is typically made up of water that may include other additives such as detergent or other treatments.

FIGS.1and2depict an exemplary domestic dishwasher or dishwasher appliance100that may be configured in accordance with aspects of the present disclosure. For the particular embodiment ofFIGS.1and2, the dishwasher100includes a cabinet102(FIG.2) having a tub104therein that defines a wash chamber106. As shown inFIG.2, tub104extends between a top107and a bottom108along a vertical direction V, between a pair of side walls110along a lateral direction L, and between a front side111and a rear side112along a transverse direction T. Each of the vertical direction V, lateral direction L, and transverse direction T are mutually perpendicular to one another.

The tub104includes a front opening114and a door116hinged at its bottom for movement between a normally closed vertical position (shown inFIG.2), wherein the wash chamber106is sealed shut for washing operation, and a horizontal, fully open position for loading and unloading of articles from the dishwasher100. A partially open position of door116are shown inFIG.1. According to exemplary embodiments, dishwasher100further includes a door closure mechanism or assembly118that is used to lock and unlock door116for accessing and sealing wash chamber106.

As best illustrated inFIG.2, tub side walls110accommodate a plurality of rack assemblies. More specifically, guide rails120may be mounted to side walls110for supporting a first rack assembly122(also referred to as a lower rack assembly122), a middle rack assembly124(also referred to as a second rack assembly124), and a third rack assembly126(also referred to as an upper rack assembly126). As illustrated, third rack assembly126is positioned at a top portion of wash chamber106above middle rack assembly124, which is positioned above lower rack assembly122along the vertical direction V. Each rack assembly122,124,126is adapted for movement between an extended loading position (not shown) in which the rack is substantially positioned outside the wash chamber106, and a retracted position (shown inFIGS.1and2) in which the rack is located inside the wash chamber106. This is facilitated, for example, by rollers128mounted onto rack assemblies122,124,126, respectively. Although a guide rails120and rollers128are illustrated herein as facilitating movement of the respective rack assemblies122,124,126, it should be appreciated that any suitable sliding mechanism or member may be used according to alternative embodiments.

Some or all of the rack assemblies122,124,126are fabricated into lattice structures including a plurality of wires or elongated members130(for clarity of illustration, not all elongated members making up rack assemblies122,124,126are shown inFIG.2). In this regard, rack assemblies122,124,126are generally configured for supporting articles within wash chamber106while allowing a flow of wash fluid to reach and impinge on those articles, e.g., during a cleaning or rinsing cycle. For this embodiment, a silverware basket (not labeled) may be removably attached to a rack assembly, e.g., lower rack assembly122, for placement of silverware, utensils, and the like, that are otherwise too small or delicate to be accommodated by rack122.

Dishwasher100further includes a plurality of spray assemblies for urging a flow of water or wash fluid onto the articles placed within wash chamber106. More specifically, as illustrated inFIG.2, dishwasher100includes a first spray assembly134(also referred to as a lower spray arm assembly134) disposed in a lower region136of wash chamber106and above a sump138so as to rotate in relatively close proximity to lower rack assembly122. Similarly, a second spray assembly140(also referred to as a mid-level spray arm assembly140) is located in an upper region of wash chamber106and may be located below and in close proximity to middle rack assembly124. In this regard, mid-level spray arm assembly140may generally be configured for urging a flow of wash fluid up through middle rack assembly124and third rack assembly126. Additionally, a third spray assembly142(also referred to as an upper spray assembly142) may be located above third rack assembly126along the vertical direction V. In this manner, third spray assembly142may be configured for urging and/or cascading a flow of wash fluid downward over rack assemblies122,124, and126.

The various spray assemblies and manifolds described herein may be part of a fluid distribution system or fluid circulation assembly150for circulating water and wash fluid in the tub104. More specifically, fluid circulation assembly150includes a pump152for circulating water and wash fluid (e.g., detergent, water, and/or rinse aid) in the tub104. Pump152may be located within sump138or within a machinery compartment located below sump138of tub104, as generally recognized in the art. Fluid circulation assembly150may include one or more fluid conduits or circulation piping for directing water and/or wash fluid from pump152to the various spray assemblies and manifolds. For example, as illustrated inFIG.2, a primary supply conduit154may extend from pump152, along rear112of tub104along the vertical direction V to supply wash fluid throughout wash chamber106.

As illustrated, primary supply conduit154is used to supply wash fluid to mid-level spray arm assembly140. In addition, a separate secondary supply conduit155may supply wash fluid to upper spray assembly142. Diverter assembly156can allow selection between spray assemblies134,140and142being supplied with wash fluid. However, it should be appreciated that according to alternative embodiments, any other suitable plumbing configuration may be used to supply wash fluid throughout the various spray manifolds and assemblies described herein.

Each spray assembly134,140,142or other spray device may include an arrangement of discharge ports or orifices for directing wash fluid received from pump152onto dishes or other articles located in wash chamber106. The arrangement of the discharge ports, also referred to as jets, apertures, or orifices, may provide a rotational force by virtue of wash fluid flowing through the discharge ports. Alternatively, spray assemblies134,140,142may be motor-driven, or may operate using any other suitable drive mechanism. Spray manifolds and assemblies may also be stationary.

Movement of the spray arm assemblies134and140and the spray from fixed manifolds like spray assembly142provides coverage of dishes, silverware, and other dishwasher contents and articles to be cleaned with a washing spray. Other configurations of spray assemblies may be used as well. For example, dishwasher100may have additional spray assemblies for cleaning silverware, for scouring casserole dishes, for spraying pots and pans, for cleaning bottles, etc. One skilled in the art will appreciate that the embodiments discussed herein are used for the purpose of explanation only and are not limitations of the present subject matter.

In operation, pump152draws wash fluid in from sump138and pumps it to a diverter assembly156, e.g., which is positioned within sump138of dishwasher appliance. Diverter assembly156may include a diverter disk (not shown) disposed within a diverter chamber158for selectively distributing the wash fluid to the spray assemblies134,140,142and/or other spray manifolds or devices. For example, the diverter disk may have a plurality of apertures that are configured to align with one or more outlet ports (not shown) at the top of diverter chamber158. In this manner, the diverter disk may be selectively rotated to provide wash fluid to the desired spray device.

According to an exemplary embodiment, diverter assembly156is configured for selectively distributing the flow of wash fluid from pump152to various fluid supply conduits, only some of which (e.g.,154and155) are illustrated inFIG.2for clarity. More specifically, diverter assembly156may include four outlet ports (not shown) for supplying wash fluid to a first conduit for rotating lower spray arm assembly134in the clockwise direction, a second conduit for rotating lower spray arm assembly134in the counterclockwise direction, a third conduit for spraying rack assembly126(shown inFIG.2) as a silverware rack, and a fourth conduit for supplying only mid-level and/or upper spray assemblies140,142. Other configurations of diverter assembly156and/or other components (e.g., valves) may be used to allow various choices in the operation of the spray assemblies134,140, and142during a cleaning cycle.

The dishwasher100is further equipped with a controller160to regulate operation of the dishwasher100. Controller160may include one or more memory devices and one or more microprocessors, such as general or special purpose microprocessors operable to execute programming instructions or micro-control code associated with a cleaning cycle. The memory may represent random access memory such as DRAM or read only memory such as ROM or FLASH. In one embodiment, the processor executes programming instructions stored in memory. The memory may be a separate component from the processor or may be included onboard within the processor. Alternatively, controller160may be constructed without using a microprocessor, e.g., using a combination of discrete analog and/or digital logic circuitry (such as switches, amplifiers, integrators, comparators, flip-flops, AND gates, and the like) to perform control functionality instead of relying upon software.

The controller160may be positioned in a variety of locations throughout dishwasher100. In the illustrated embodiment, the controller160may be located within a control panel area162of door116as shown inFIGS.1and2. In such an embodiment, input/output (“I/O”) signals may be routed between the control system and various operational components of dishwasher100along wiring harnesses that may be routed through the bottom of door116. Typically, the controller160includes a user interface panel/controls164through which a user may select various operational features and modes and monitor progress of the dishwasher100. In one embodiment, the user interface164may represent a general purpose I/O (“GPIO”) device or functional block. In one embodiment, the user interface164may include input components, such as one or more of a variety of electrical, mechanical or electro-mechanical input devices including rotary dials, push buttons, and touch pads. The user interface164may include a display component, such as a digital or analog display device designed to provide operational feedback to a user. The user interface164may be in communication with the controller160via one or more signal lines or shared communication busses.

It should be appreciated that the invention is not limited to any particular style, model, or configuration of dishwasher100. The exemplary embodiment depicted inFIGS.1and2is for illustrative purposes only. For example, different locations may be provided for user interface164, different configurations may be provided for rack assemblies122,124,126, different spray assemblies134,140,142and spray manifold configurations may be used, and other differences may be applied while remaining within the scope of the present subject matter.

Dishwasher100may also be configured to communicate wirelessly with a cloud-server that may include a database or may be, e.g., a cloud-based data storage system and may also include image recognition and processing capabilities including artificial intelligence as further described below. For example, appliance100may communicate with cloud-server over the Internet, and appliance100may access via WI-FI®, such as from a WI-FI® access point in a user's home or through a mobile device. Alternatively, dishwasher100may be equipped with such image recognition and processing capabilities as part of controller160and/or other components onboard appliance100.

In this regard, referring still toFIG.1, a schematic diagram of an external communication system170will be described according to an exemplary embodiment of the present subject matter. In general, external communication system170is configured for permitting interaction, data transfer, and other communications between dishwasher appliance100and one or more external devices. For example, this communication may be used to provide and receive operating parameters, user instructions or notifications, performance characteristics, user preferences, or any other suitable information for improved performance of dishwasher appliance100. In addition, it should be appreciated that external communication system170may be used to transfer data or other information to improve performance of one or more external devices or appliances and/or improve user interaction with such devices.

For example, external communication system170permits controller160of dishwasher appliance100to communicate with a separate device external to dishwasher appliance100, referred to generally herein as an external device172. As described in more detail below, these communications may be facilitated using a wired or wireless connection, such as via a network174. In general, external device172may be any suitable device separate from dishwasher appliance100that is configured to provide and/or receive communications, information, data, or commands from a user. In this regard, external device172may be, for example, a personal phone, a smartphone, a tablet, a laptop or personal computer, a wearable device, a smart home system, or another mobile or remote device.

In addition, a remote server176may be in communication with dishwasher appliance100and/or external device172through network174. In this regard, for example, remote server176may be a cloud-based server176, and is thus located at a distant location, such as in a separate state, country, etc. According to an exemplary embodiment, external device172may communicate with a remote server176over network174, such as the Internet, to transmit/receive data or information, provide user inputs, receive user notifications or instructions, interact with or control dishwasher appliance100, etc. In addition, external device172and remote server176may communicate with dishwasher appliance100to communicate similar information.

In general, communication between dishwasher appliance100, external device172, remote server176, and/or other user devices or appliances may be carried using any type of wired or wireless connection and using any suitable type of communication network, non-limiting examples of which are provided below. For example, external device172may be in direct or indirect communication with dishwasher appliance100through any suitable wired or wireless communication connections or interfaces, such as network174. For example, network174may include one or more of a local area network (LAN), a wide area network (WAN), a personal area network (PAN), the Internet, a cellular network, any other suitable short- or long-range wireless networks, etc. In addition, communications may be transmitted using any suitable communications devices or protocols, such as via Wi-Fi®, Bluetooth®, Zigbee®, wireless radio, laser, infrared, Ethernet type devices and interfaces, etc. In addition, such communication may use a variety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and/or protection schemes (e.g., VPN, secure HTTP, SSL).

External communication system170is described herein according to an exemplary embodiment of the present subject matter. However, it should be appreciated that the exemplary functions and configurations of external communication system170provided herein are used only as examples to facilitate description of aspects of the present subject matter. System configurations may vary, other communication devices may be used to communicate directly or indirectly with one or more associated appliances, other communication protocols and steps may be implemented, etc. These variations and modifications are contemplated as within the scope of the present subject matter.

Dishwasher100includes a camera assembly or other optical sensor assembly180, which may be positioned along one of the sidewalls110. For this exemplary embodiment, camera assembly180includes a sensor182(e.g., a camera) for obtaining images within wash chamber106and particularly images that can be used to monitor the wash fluid (e.g., identified generally by reference numeral184inFIGS.4and5). In other embodiments, camera assembly180may be positioned so that it has a view of another desired location in, or desired portion of, wash chamber106. Thus, although shown on one of the sidewalls110near door116, camera assembly180could be placed at other locations along sidewalls110, rear wall112, or even door116provided such placement allows for a view and resulting capture of an image from which the wash fluid184may be monitored.

Camera assembly180may include any suitable number, type, size, and configuration of camera(s)182for obtaining images in wash chamber106. In general, camera(s)182may include a lens that is constructed from a clear hydrophobic material or which may otherwise be positioned behind a hydrophobic clear lens. So positioned, camera assembly180may obtain one or more images or videos of articles and/or rack assemblies within wash chamber106, as described in more detail below. For the exemplary embodiment ofFIG.2, camera assembly180is positioned in view of first rack assembly122where a silverware basket would normally be located and/or where lower spray arm assembly134would be supplying wash fluid184.

Referring still toFIG.2, dishwasher appliance100may further include one or more wash chamber lights186positioned within cabinet102or wash chamber106for selectively illuminating wash chamber106, the articles positioned therein, and/or the wash fluid184. Specifically, light186may be separate from camera assembly180or may be integrated into camera assembly180. In one embodiment, light186is positioned immediately adjacent camera assembly180. According to still other embodiments, light186may be positioned at any other suitable location within cabinet102. It should be appreciated that according to alternative embodiments, dishwasher appliance100may include any other camera or system of imaging devices for obtaining images to monitor the flow of wash fluid184. In addition, these cameras may be positioned at any suitable location within cabinet102, may include any suitable lighting features, and may utilize any suitable photography or imaging technology.

Notably, controller160of dishwasher appliance100(or any other suitable dedicated controller) may be communicatively or operably coupled to camera assembly180, camera182, tub light186, and/or other components of appliance100. As explained in more detail below, controller160may be programmed or configured for analyzing the images obtained by camera assembly180, e.g., in order to monitor the flow of wash fluid184and may use this information to make informed decisions regarding the operability of spray arm assemblies134,140,142or the operation of fluid circulation assembly150. Alternatively, such images from camera assembly180may be transmitted or uploaded to e.g., a cloud-server or cloud-based system (e.g., remote server176) for further processing of such information as will also be further described. The images may also be electronically stored by dishwasher appliance100as part of the process by which dishwasher appliance100utilizes the same to monitor the wash fluid184.

Referring now toFIG.3, an exemplary method200of operating a dishwashing appliance such as dishwasher appliance100will be described. Although the discussion below refers to the exemplary method200of operating appliance100, one skilled in the art will appreciate that the exemplary method200is applicable to the operation of a variety of other dishwashing appliances having different configurations and equipment and that the steps disclosed herein may be performed by e.g., controller160in whole, or in part, and in conjunction with one or more separate systems including cloud-based systems. Reference to a “step” or other action does not prevent such from being performed in a series of steps or multiple actions unless otherwise stated and does mean such step is necessarily required in all exemplary aspects of the present invention. The order of such steps or actions may also be altered unless otherwise stated.

In step210, the exemplary algorithm or method200is initiated. One or more of a variety of events might cause e.g., controller160to begin execution of the steps in method200. For example, step210could be that dishwasher100is activated or started. For example, through interface164, a user may actuate a cleaning cycle of appliance100. Such may include the selection, using interface164, of one or more options for the cleaning cycle followed by closing door116. Activation may also come after a period of delay, which the user may select. In still other examples, the “start” in step210could be the powering up of appliance100after which method200proceeds to step220.

Next, in step220, method200may include determining an operating state of a fluid delivery device (e.g., fluid circulation system150). For example, step220may include detecting the operational status of pump152, the position of diverter assembly156, etc. Step220may further include identifying, based on the operating state of the fluid circulation assembly, which spray arms should be activated and actively spraying to facilitate a cleaning cycle. For example, to facilitate discussion of aspects of the present subject matter, the remainder of the discussion of method200will assume that lower spray arm assembly134is activated for the cleaning cycle. However, it should be appreciated that aspects of the present disclosure are equally applicable to methods of monitoring the operation of any other spray arm or fluid dispensing device within any suitable dishwasher.

Step230include obtaining or capturing one or more images (which may be e.g., still shots, videos, or both) of using camera assembly180. For example, according to an example embodiment, step230may include taking consecutive images, e.g., such as a time lapse of two or more images for comparison. As used herein, “image” includes a single photograph or representation (e.g., a digital or electronic file) of the view of camera assembly180, multiple such photographs or representations, and/or videos from which image processing can be performed to monitor wash fluid184within wash chamber106.

In this regard, referring briefly toFIGS.4and5, various images obtained during the performance of method200are provided for purposes of facilitating explanation of aspects of the present subject matter. Although two consecutive images are illustrated in each figure, it should be appreciated that any suitable number of images may be used while remaining within the scope of the present subject matter. Moreover, these images may be obtained with or without light186, from any suitable position of camera182, at any suitable frequency and resolution, etc. As explained herein, aspects of the present subject matter are generally directed toward the use of such obtained images to determine whether fluid circulation assembly150is operating properly, or more specifically, whether one or more spray arms134,140,142are blocked (e.g., prevented from rotating or positioned behind an object blocking some or all of their spray).

For example,FIG.4illustrates a first image300and a second image302obtained by camera assembly180of wash chamber106when the fluid circulation assembly150is operating properly and no spray arms are blocked. The first image300and second image302may be obtained consecutively, e.g., within a few seconds of each other or at any suitable frequency. By contrast,FIG.5illustrates a third image304and a fourth image306obtained by camera assembly180of wash chamber106when the fluid circulation assembly150is not operating properly, i.e., where at least one spray arm is not rotating, blocked, or otherwise not operating properly. The third image304and fourth image306may be obtained consecutively, e.g., within a few seconds of each other or at any suitable frequency

Step240generally includes uploading the images obtained at step230(e.g., one or more of images300-306) to an artificial intelligence (“AI”) server or module on the cloud or a dedicated module located within dishwasher appliance100(e.g., on controller160). In this regard, controller160may obtain the images300-306using camera assembly180and may either analyze those images300-306using an on-board AI module or may offload such analysis to a remote module. This remote module may be located elsewhere within dishwasher appliance100or remote from appliance (e.g., on remote server176). The remainder of the discussion of method200refers to image analysis being performed on the cloud (e.g., on remote server176). However, it should be appreciated that this analysis could alternatively be performed locally on the dishwasher appliance100or at any other suitable location.

Step250may generally include determining whether the fluid delivery device (e.g., fluid circulation system150) is providing acceptable delivery of the wash fluid into the wash chamber. For example, this determination may include using the consecutive images obtained at step230to determine whether there are differences in the wash fluid184being sprayed within the wash chamber106. In other words, step250may determine whether there is a substantially change in the flow of wash fluid184between consecutive images.

In addition, or alternatively, determining whether the fluid delivery device is operating properly may include comparing one or more of the consecutive images to one or more baseline reference images, wherein the baseline reference images include images representative of a targeted presence of the wash fluid in the wash chamber. In this regard, controller160may be supplied with a plurality of baseline images showing the flow of wash fluid184within wash chamber106when a spray arm is blocked and/or not blocked. By comparing the obtained images with the baseline images, controller160may determine whether the fluid delivery device is operating more like that shown in the baseline images where a spray arm is blocked (e.g., the fluid delivery device is not operating properly) or more like that shown in the baseline images where a spray arm is not blocked (e.g., the fluid delivery device is operating properly).

As explained in more detail below, various methods of image analysis may be used to make this determination at step250. However, aspects of the present subject matter are directed to identifying patterns in the flow of wash fluid184within wash chamber106to determine whether fluid circulation assembly150is operating properly or whether a spray arm134,140,142is blocked. Although exemplary wash fluid signatures or patterns are described herein as being associated with a blocked (or unblocked) spray arm, it should be appreciated that other signatures/patterns may be used while remaining within the scope of the present subject matter. Indeed, the AI module may be trained with numerous images of various spray arms in both the blocked and unblocked conditions to train the AI module to properly identify the present spray arm condition in any situation.

For example, referring again briefly toFIGS.4and5,FIG.4provides example consecutive images300,302of an unblocked (e.g., properly operating) lower spray arm assembly134. As shown, the wash fluid184circulating or spraying within wash chamber106is flowing in various streams of water (e.g., identified by reference numeral310) and various splashes or droplets (e.g., identified by reference numeral312) that are either deflected off objects or are dripping down from the upper racks. As shown, the wash fluid184between images300and302is largely random, e.g., the streams of water310are clearly moving between images and there are a large number of droplets312which are relatively uniformly dispersed throughout wash chamber106. This may be indicative of a properly operating fluid circulation assembly150supplying well distributed wash fluid184.

By contrast,FIG.5provides example consecutive images304,306of a blocked (e.g., improperly operating) lower spray arm assembly134. As shown, the wash fluid184circulating or spraying within wash chamber106is flowing in various streams of water310that are largely uniform between images304and306. In addition, the various splashes or droplets312are localized or not as widely dispersed as those fromFIG.4. The streams310and droplets312are not random, but instead seem to be substantially constant between images, which may be indicative of an improperly operating fluid circulation assembly150. Aspects of the present subject matter, such as those implemented in step250, are generally directed to identifying the operational state of the fluid circulation assembly150by identifying these (and other) characteristics from images obtained by camera assembly180. As explained, other water distribution patterns or signatures may be identified and included in baseline images for comparison and image analysis.

Specifically, the analysis of the one or more images may include implementation an image processing algorithm. As used herein, the terms “image processing” and the like are generally intended to refer to any suitable methods or algorithms for analyzing images that do not rely on artificial intelligence or machine learning techniques (e.g., in contrast to the machine learning image recognition processes described below). For example, the image processing algorithm may rely on image differentiation, e.g., such as a pixel-by-pixel comparison of two sequential images. This comparison may help identify substantial differences between the sequentially obtained images, e.g., to identify movement, the presence of a particular object, the existence of a certain condition, etc. For example, one or more reference images may be obtained when a particular condition exists, and these references images may be stored for future comparison with images obtained during appliance operation. Similarities and/or differences between the reference image and the obtained image may be used to extract useful information for improving appliance performance.

According to exemplary embodiments, image processing may include blur detection algorithms that are generally intended to compute, measure, or otherwise determine the amount of blur in an image. For example, these blur detection algorithms may rely on focus measure operators, the Fast Fourier Transform along with examination of the frequency distributions, determining the variance of a Laplacian operator, or any other methods of blur detection known by those having ordinary skill in the art. In addition, or alternatively, the image processing algorithms may use other suitable techniques for recognizing or identifying items or objects, such as edge matching or detection, divide-and-conquer searching, greyscale matching, histograms of receptive field responses, or another suitable routine (e.g., executed at the controller160based on one or more captured images from one or more cameras). Other image processing techniques are possible and within the scope of the present subject matter. The processing algorithm may further include measures for isolating or eliminating noise in the image comparison, e.g., due to image resolution, data transmission errors, inconsistent lighting, or other imaging errors. By eliminating such noise, the image processing algorithms may improve accurate object detection, avoid erroneous object detection, and isolate the important object, region, or pattern within an image.

In addition to the image processing techniques described above, the image analysis may include utilizing artificial intelligence (“AI”), such as a machine learning image recognition process, a neural network classification module, any other suitable artificial intelligence (AI) technique, and/or any other suitable image analysis techniques, examples of which will be described in more detail below. Moreover, each of the exemplary image analysis or evaluation processes described below may be used independently, collectively, or interchangeably to extract detailed information regarding the images being analyzed to facilitate performance of one or more methods described herein or to otherwise improve appliance operation. According to exemplary embodiments, any suitable number and combination of image processing, image recognition, or other image analysis techniques may be used to obtain an accurate analysis of the obtained images.

In this regard, the image recognition process may use any suitable artificial intelligence technique, for example, any suitable machine learning technique, or for example, any suitable deep learning technique. According to an exemplary embodiment, the image recognition process may include the implementation of a form of image recognition called region based convolutional neural network (“R-CNN”) image recognition. Generally speaking, R-CNN may include taking an input image and extracting region proposals that include a potential object or region of an image. In this regard, a “region proposal” may be one or more regions in an image that could belong to a particular object or may include adjacent regions that share common pixel characteristics. A convolutional neural network is then used to compute features from the region proposals and the extracted features will then be used to determine a classification for each particular region.

According to still other embodiments, an image segmentation process may be used along with the R-CNN image recognition. In general, image segmentation creates a pixel-based mask for each object in an image and provides a more detailed or granular understanding of the various objects within a given image. In this regard, instead of processing an entire image—i.e., a large collection of pixels, many of which might not contain useful information—image segmentation may involve dividing an image into segments (e.g., into groups of pixels containing similar attributes) that may be analyzed independently or in parallel to obtain a more detailed representation of the object or objects in an image. This may be referred to herein as “mask R-CNN” and the like, as opposed to a regular R-CNN architecture. For example, mask R-CNN may be based on fast R-CNN which is slightly different than R-CNN. For example, R-CNN first applies a convolutional neural network (“CNN”) and then allocates it to zone recommendations on the covn5 property map instead of the initially split into zone recommendations. In addition, according to exemplary embodiments, standard CNN may be used to obtain, identify, or detect any other qualitative or quantitative data related to one or more objects or regions within the one or more images. In addition, a K-means algorithm may be used.

According to still other embodiments, the image recognition process may use any other suitable neural network process while remaining within the scope of the present subject matter. For example, the step of analyzing the one or more images may include using a deep belief network (“DBN”) image recognition process. A DBN image recognition process may generally include stacking many individual unsupervised networks that use each network's hidden layer as the input for the next layer. According to still other embodiments, the step of analyzing one or more images may include the implementation of a deep neural network (“DNN”) image recognition process, which generally includes the use of a neural network (computing systems inspired by the biological neural networks) with multiple layers between input and output. Other suitable image recognition processes, neural network processes, artificial intelligence analysis techniques, and combinations of the above described or other known methods may be used while remaining within the scope of the present subject matter.

In addition, it should be appreciated that various transfer techniques may be used but use of such techniques is not required. If using transfer techniques learning, a neural network architecture may be pretrained such as VGG16/VGG19/ResNet50 with a public dataset then the last layer may be retrained with an appliance specific dataset. In addition, or alternatively, the image recognition process may include detection of certain conditions based on comparison of initial conditions, may rely on image subtraction techniques, image stacking techniques, image concatenation, etc. For example, the subtracted image may be used to train a neural network with multiple classes for future comparison and image classification.

It should be appreciated that the machine learning image recognition models may be actively trained by the appliance with new images (e.g., baseline images), may be supplied with training data from the manufacturer or from another remote source, or may be trained in any other suitable manner. For example, according to exemplary embodiments, this image recognition process relies at least in part on a neural network trained with a plurality of images of the appliance in different configurations, experiencing different conditions, or being interacted with in different manners. This training data may be stored locally or remotely and may be communicated to a remote server for training other appliances and models. According to exemplary embodiments, it should be appreciated that the machine learning models may include supervised and/or unsupervised models and methods. In this regard, for example, supervised machine learning methods (e.g., such as targeted machine learning) may help identify problems, anomalies, or other occurrences which have been identified and trained into the model. By contrast, unsupervised machine learning methods may be used to detect clusters of potential failures, similarities among data, event patterns, abnormal concentrations of a phenomenon, etc.

It should be appreciated that image processing and machine learning image recognition processes may be used together to facilitate improved image analysis, object detection, or to extract other useful qualitative or quantitative data or information from the one or more images that may be used to improve the operation or performance of the appliance. Indeed, the methods described herein may use any or all of these techniques interchangeably to improve image analysis process and facilitate improved appliance performance and consumer satisfaction. The image processing algorithms and machine learning image recognition processes described herein are only exemplary and are not intended to limit the scope of the present subject matter in any manner.

Step260may include determining whether an obstruction or malfunction of fluid circulation assembly150is detected. In this regard, based on the image analysis and comparison performed at step250, controller160may conclude that lower spray arm134(or another spray arm) is blocked or is not blocked. In the event the spray arm is not blocked, step270may include continuing the dishwasher cycle as normal.

By contrast, if step260results in a determination that one or more spray arms is blocked, step280may include implementing a corrective action, e.g., because the fluid delivery device is not providing an acceptable delivery of the wash fluid into the wash chamber. For example, step280may include providing a user notification in response to identifying an obstructed spray arm. For example, step280may include sounding an alarm (e.g., via control panel162) or providing some other suitable indication on an appliance display. According to still other embodiments, the user notification may be provided directly to the user through a remote device172(e.g., such as through a software application on the user's cell phone) over network174. According to exemplary embodiments, this user notification may provide a user with details related to the condition detected, the spray arm obstructed, recommended procedures for correcting, etc. Implementing the corrective action may further include adjusting an operating parameter (e.g., such as operating only upper spray arms) or stopping operation of dishwasher appliance100altogether until the issue is rectified.

FIG.3depicts steps performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the steps of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, or modified in various ways without deviating from the scope of the present disclosure. Moreover, although aspects of method200are explained using dishwasher appliance100as an example, it should be appreciated that this method may be applied to the operation of any suitable dishwasher appliance.

As explained herein, aspects of the present subject matter are generally directed to a dishwasher having a camera to verify that one or more spray arms are not blocked. For example, the camera may be used to monitor the lower spray arm to determine whether it is blocked/not blocked through the motion of water from lower spray arm. In specific, the artificial intelligence technology may use the motion of the water that the lower spray arm produces to conclude if it is blocked or not. In this regard, when the consumer starts the cycle, the diverter may be in the lower spray arm position and circulation pump is activated. The camera may take a series of pictures (time-lapse) of the motion of water in the lower portion of the dishwasher. The controller may upload the time lapse pictures to a connected dishwasher AI server on the cloud or on the unit itself. The AI server on cloud/unit determines may determine if the lower spray arm is blocked by dishware or otherwise not properly operating. The connected dishwasher AI server on cloud/unit may send data to the dishwasher control, and if an obstruction is detected, a fault code may be recorded. In addition, the dishwasher cycle may be paused up to predetermined number of minutes and an alert may be sent to the consumer to correct the issue via a software application or a user interface.

This written description uses examples to disclose the present disclosure, including the best mode, and also to enable any person skilled in the art to practice the disclosure, including making and using any devices or systems and performing any incorporated methods. The patentable scope of the disclosure is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they include structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal languages of the claims.