Abstract:
Systems and methods for boil detection are provided. One example system includes a vision sensor positioned to collect imagery depicting the cooktop and one or more processors implementing instructions to perform operations. The operations include obtaining a first frame of imagery and a second frame of imagery from the vision sensor. The first frame of imagery and the second frame of imagery respectively depict the cooktop. The operations include identifying a plurality of motion vectors based on the first and second frames of imagery. The operations include generating a histogram describing a characteristic of the plurality of motion vectors. The operations include calculating a dissimilarity score. The dissimilarity score describes a difference between the histogram and a pre-learned histogram. The pre-learned histogram describes the characteristic for imagery depicting boiling liquid. The operations include determining whether a boiling event is occurring at the cooktop based at least in part on the dissimilarity score.

Description:
FIELD OF THE INVENTION 
     The present disclosure relates generally to systems and methods for boil detection. More particularly, the present disclosure is directed to systems and methods for boil detection in which a computer vision algorithm is applied to imagery of a cooktop captured by a camera to detect a boiling event. 
     BACKGROUND OF THE INVENTION 
     One primary use of a cooktop appliance is to boil water or other liquids. For example, when preparing rice, pasta, or other dishes, it may be desirable to bring a pot of water to boil using high heat and then reduce the level of supplied heat once the water begins to boil. As another example, some dishes may require bringing a sauce to boil and then reducing heat to let the sauce simmer. 
     However, in certain instances, the user may be distracted or performing another task and, therefore, fail to notice that the water or other liquid has reached a boiling state. Failure to reduce the heat once the liquid has achieved the boiling state can cause a number of problems including, for example, overcooking of the dish, splatter of liquid onto the cooktop surface, or even complete evaporation of the liquid, a condition referred to as “boil-dry,” which can potentially lead to ignition of a fire. Therefore, systems and methods for boil detection and alarming are desirable. 
     Certain existing systems have been proposed for performing boil detection. As an example, motion sensors can be used to detect motion at the cooktop. However, these systems suffer from significant problems with accuracy, as human motion (e.g. stirring) or rising steam trigger the sensor and leads to a false positive of a boiling event. 
     As another example, other existing systems may use temperature sensors to attempt to detect a boiling event. However, these systems can suffer from problems with accuracy and granularity, as well. For example, temperature sensors in a generally heated environment such as a cooktop may lead to significant numbers of errors. 
     Therefore, systems and methods for boil detection that provide improved accuracy are desirable. 
     BRIEF DESCRIPTION OF THE INVENTION 
     Additional aspects and advantages of the invention will be set forth in part in the following description, or may be apparent from the description, or may be learned through practice of the invention. 
     One aspect of the present disclosure is directed to a system for detecting a boiling event at a cooktop. The system includes a vision sensor positioned so as to collect imagery depicting the cooktop. The system includes one or more processors and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. The operations include obtaining a first frame of imagery and a second frame of imagery from the vision sensor. The first frame of imagery and the second frame of imagery respectively depict the cooktop. The operations include identifying a plurality of motion vectors based on the first and second frames of imagery. Each of the plurality of motion vectors describes a change in position of one or more pixels from the first frame of imagery to the second frame of imagery. The operations include generating a histogram describing a characteristic of the plurality of motion vectors. The operations include calculating a dissimilarity score. The dissimilarity score describes a difference between the histogram and a pre-learned histogram. The pre-learned histogram describes the characteristic for imagery depicting boiling liquid. The operations include determining whether a boiling event is occurring at the cooktop based at least in part on the dissimilarity score. 
     Another aspect of the present disclosure is directed to a method for detecting a boiling event at a cooktop. The method includes collecting, by a vision sensor, a plurality of frames comprising imagery depicting the cooktop. The method includes determining, by one or more computing devices, a plurality of motion vectors for each of the plurality of frames. Each motion vector for each frame describes a change in position of one or more pixels included in such frame with respect to a previous sequential frame of the plurality of frames. The method includes generating, by the one or more computing devices for each of the plurality of frames, one or more histograms describing the plurality of motion vectors determined for such frame. The method includes comparing, by the one or more computing devices for each of the plurality of frames, the one or more histograms generated for such frame with one or more pre-learned histograms. The method includes detecting, by the one or more computing devices, a boiling event at the cooktop based on the comparison of the one or more histograms generated for each frame with the one or more pre-learned histograms. 
     Another aspect of the present disclosure is directed to a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations. The operations include obtaining a plurality of frames of imagery from a vision sensor. The plurality of frames of imagery respectively depict a cooktop. The plurality of frames of imagery form a plurality of pairs of consecutive frames. The operations include identifying a plurality of motion vectors for each pair of consecutive frames of imagery. The operations include generating, for each pair of consecutive frames of imagery, a histogram describing a characteristic of the plurality of motion vectors identified for such pair of consecutive frames. The operations include calculating, for each pair of consecutive frames of imagery, a dissimilarity score. The dissimilarity score describes a difference between the histogram generated for such pair of consecutive frames and a pre-learned histogram. The pre-learned histogram describes the characteristic for imagery depicting boiling liquid. The operations include determining whether a boiling event is occurring at the cooktop based at least in part on the dissimilarity scores respectively calculated for the plurality of pairs of consecutive frames. 
     These and other features, aspects and advantages of the present invention will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       A full and enabling disclosure of the present invention, including the best mode thereof, directed to one of ordinary skill in the art, is set forth in the specification, which makes reference to the appended figures, in which: 
         FIG. 1  depicts an example system according to an example embodiment of the present disclosure; 
         FIG. 2  depicts a flow chart of an example method according to an example embodiment of the present disclosure; 
         FIG. 3  depicts an example depiction of a vision sensor capturing imagery depicting a cooktop according to an example embodiment of the present disclosure; 
         FIG. 4  depicts an example frame of imagery depicting a cooktop according to an example embodiments of the present disclosure; 
         FIG. 5  depicts an example frame of imagery depicting a cooktop according to an example embodiments of the present disclosure; 
         FIG. 6  depicts an example histogram according to an example embodiment of the present disclosure; 
         FIG. 7  depicts an example histogram according to an example embodiment of the present disclosure; 
         FIG. 8  depicts an example histogram according to an example embodiment of the present disclosure; and 
         FIG. 9  depicts an example histogram according to an example embodiment of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     Reference now will be made in detail to embodiments of the invention, one or more examples of which are illustrated in the drawings. Each example is provided by way of explanation of the invention, not limitation of the invention. In fact, it will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the scope or spirit of the invention. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present invention covers such modifications and variations as come within the scope of the appended claims and their equivalents. 
     With reference now to the FIGS., example embodiments of the present disclosure will be discussed in further detail.  FIG. 1  depicts an example system  100  according to an example embodiment of the present disclosure. 
     System  100  can include a vision sensor  102  that can exchange data with a computing device  104 . As an example, the vision sensor  102  can be any suitable camera for capturing imagery. For example, in some embodiments, the vision sensor  102  can be a wide-angle VGA resolution camera. 
     vision sensor  102  can be positioned so as to collect imagery depicting a cooktop. For example, vision sensor  102  can be secured to the underside of an over the range microwave or hood and pointed downwards so that the view of the vision sensor generally corresponds to the dimensions of the cooktop. 
     Vision sensor  102  can collect a plurality of frames of imagery. For example, in some embodiments, computing device  104  can operate vision sensor  102  to collect between 0.3 to 1 frames per second of a VGA resolution video stream. The frame rate can be modifiable by the computing device  104 . 
     Computing device  104  can be any device that includes one or more processors  106  and a memory  108 . As an example, in some embodiments, computing device  104  can be a single board computer (SBC). For example, the computing device  104  can be a single System-On-Chip (SOC). Further, the vision sensor  102  can also be located on the same single circuit board. However, any form of computing device  104  can be used to perform the present disclosure. 
     The processor(s)  106  can be any suitable processing device, such as a microprocessor, microcontroller, integrated circuit, or other suitable processing devices or combinations thereof. 
     The memory  108  can include any suitable storage media, including, but not limited to, non-transitory computer-readable media, RAM, ROM, hard drives, flash drives, accessible databases, or other memory devices. The memory  108  can store information accessible by processor(s)  106 , including instructions  110  that can be executed by processor(s)  106  to perform aspects of the present disclosure. 
     Memory  108  can also include a counter  112  storing a counter value. In some embodiments, the counter  112  can be used to count a number of consecutive frames of imagery which the computing device  104  has determined depict a boiling event at the cooktop. 
     Computing device  104  can also include a number of modules to provide functionality or otherwise perform particular operations. It will be appreciated that the term “module” refers to computer logic utilized to provide desired functionality. Thus, a module can be implemented in hardware, application specific circuits, firmware and/or software controlling a general purpose processor. In one embodiment, modules are program code files stored on the storage device, loaded into memory and executed by a processor or can be provided from computer program products, for example computer executable instructions, that are stored in a tangible computer-readable storage medium such as RAM, hard disk or optical or magnetic media. 
     Computing device  104  can implement a motion vector determination module  114  to determine a plurality of motion vectors for each of a plurality of pairs of frames of imagery. For example, each of the plurality of motion vectors can describe a change in position of one or more pixels from a first frame of imagery to a second frame of imagery. In some embodiments, each motion vector can include a magnitude and an orientation. For example, the orientation can be expressed as an angle from 0 to 360. 
     In particular, in some embodiments, motion vector determination module  114  can be implemented to perform one or more motion estimation techniques to identify the plurality of motion vectors. As examples, motion vector determination module  114  can perform a block-matching technique, a phase correlation technique, a frequency domain technique, a pixel recursive technique, an optical flow technique (e.g. Lucas-Kanade or Horn-Schunck), corner detection, feature matching, patch matching, texture matching, or other techniques or statistical functions. 
     As another example, in some embodiments, processing performed by motion vector determination module  114  can be simplified by searching only along horizontal and/or vertical lines extending from each pixel in the earlier frame for matching pixels in the later frame. In such fashion, processing time and power requirements can be reduced to allow for an improved implementation speed or reduced product cost. 
     Computing device  104  can implement a histogram generation module  116  to generate one or more histograms for each of the plurality of pairs of frames of imagery. For example, each histogram can describe a characteristic of the plurality of motion vectors determined for each respective pair of frames. As examples, the histogram can describe a distribution of orientations associated with the plurality of motion vectors; a distribution of magnitudes associated with the plurality of motion vectors; a distribution of other variables; or distributions of various combinations of variables (e.g. both orientation and magnitude). The histograms may be embodied as stored data and are not necessarily represented or stored in a graphical or visual fashion. Further, the histograms can provide discrete intervals of data or may provide a continuous density function with kernel methods. 
     Computing device  104  can implement a histogram comparison module  118  to compare two or more histograms. For example, histogram comparison module  118  can be implemented to compare the one or more histograms generated by histogram generation module  116  with one or more pre-learned histograms stored in memory  108 . 
     In particular, the one or more pre-learned histograms stored in memory  108  can respectively describe one or more characteristics of a plurality of motion vectors exhibited by frames of imagery depicting a boiling event. For example, in some embodiments, the pre-learned histograms can be generated through testing or analyzing imagery depicting a boiling event. In other embodiments, the pre-learned histograms can be approximations, such as for example, a histogram having a uniform distribution of orientations. 
     Histogram comparison module  118  can be implemented to perform one or more histogram comparison techniques to compare two or more histograms. As examples, histogram comparison module  118  can perform an earth mover&#39;s distance (EMD) technique, a bin-to-bin comparison technique, an L-1 distance technique, an L-2 distance technique, a Wasserstein metric technique, a Kolmogorov-Smirnov test, an f-divergence technique (e.g. Kullback-Leiber divergence or Hellinger distance), a total variation distance technique, an absolute difference value, or other histogram comparison techniques. 
     Furthermore, histogram comparison module  118  can provide a dissimilarity score for two histograms being compared. As an example, the dissimilarity score can correspond to one or more distance values that respectively result from one or more histogram comparison techniques. For example, the dissimilarity score can be a weighted sum of a plurality of distance values respectively obtained using a plurality of histogram comparison techniques. In some embodiments, a smaller dissimilarity score will be provided for pairs of histograms that exhibit a similar distribution while larger dissimilarity score will be provided for pairs of histograms that exhibit divergent distributions. 
     Alarm means  120  can be any suitable component or grouping of components for providing an alarm to a user upon detection of a boiling event. As an example, alarm means  120  can include one or more light emitting devices (e.g. LEDs) that illuminate or flash upon detection of the boiling event or one or more speakers for providing an audible alarm upon detection of the boiling event. 
     As another example, in some embodiments, alarm means  120  can include circuitry for communicating with the cooktop appliance over a local network to inform the cooktop appliance of the boiling event. In such instance, the cooktop appliance may respond by lowering the temperature setting for the burner associated with the boiling event. 
     As yet another example, in some embodiments, alarm means  120  can include circuitry for sending an SMS text message or other form of electronic message (e.g. electronic mail) to a mobile computing device of a user to notify the user of the detected boiling event. 
       FIG. 2  depicts a flow chart of an example method ( 200 ) according to an example embodiment of the present disclosure. Although method ( 200 ) will be discussed with reference to system  100  of  FIG. 1 , method ( 200 ) can be performed by any suitable system. In addition,  FIG. 2  depicts steps performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the various steps of method ( 200 ) can be omitted, adapted, and/or rearranged in various ways without departing from the scope of the present disclosure. 
     At ( 202 ) a frame of imagery depicting a cooktop can be obtained. For example, computing device  104  can obtain a frame of imagery from vision sensor  102 . As an example,  FIG. 3  depicts an example depiction of a vision sensor capturing imagery depicting a cooktop according to an example embodiment of the present disclosure. 
     Referring again to  FIG. 2 , at ( 204 ) a plurality of motion vectors can be determined for the frame with respect to a previous sequential frame. For example, computing device  104  can implement motion vector determination module  114  to determine the plurality of motion vectors for the frame. 
     As an example,  FIG. 4  depicts an example frame  400  of imagery depicting a cooktop according to an example embodiment of the present disclosure. In particular, frame  400  depicts a pot of water that is not boiling and a human that is moving. 
     Furthermore, a plurality of motion vectors are superimposed graphically upon frame  400 . For example, motion vector  402  shows motion of the human arm while motion vector  404  shows motion of the human leg. As can be seen from  FIG. 4 , the motion vectors showing human motion in frame  400  are generally relatively large in magnitude and consistent in orientation. 
     As another example,  FIG. 5  depicts an example frame  500  f imagery depicting a cooktop according to an example embodiment of the present disclosure. In particular, frame  500  depicts a pot of water that is boiling. 
     Furthermore, a plurality of motion vectors are superimposed graphically upon frame  500 . For example, motion vector  502  shows motion captured by frame  500  that corresponds to the boiling water. As can be seen from  FIG. 5 , the motion vectors showing motion due to boiling water in frame  500  are generally relatively small in magnitude and randomized in orientation. 
     Referring again to  FIG. 2 , at ( 206 ) one or more histograms can be generated that describe one or more characteristics of the plurality of motion vectors determined at ( 204 ). For example, computing device  106  can implement histogram generation module  116  to generate the one or more histograms. 
     As an example,  FIG. 6  depicts an example histogram  600  according to an example embodiment of the present disclosure. Histogram  600  describes a distribution of motion vector orientations corresponding to the plurality of motion vectors shown on frame  400  of  FIG. 4 , which depicts human motion. In particular, because the motion vectors for frame  400  are relatively consistent in orientation (e.g. showing the arm movement in a first direction and the leg movement in the second direction), the corresponding histogram  600  reflects a disproportionate number of motion vectors at two particular orientations. 
     As another example,  FIG. 7  depicts an example histogram  700  according to an example embodiment of the present disclosure. Histogram  700  describes a distribution of motion vector orientations corresponding to the plurality of motion vectors shown on frame  500  of  FIG. 5 , which depicts a boiling event. In particular, because the motion vectors for frame  500  are relatively randomized in orientation, the corresponding histogram  700  reflects a relatively uniform distribution of motion vector orientations. 
     Thus, if a histogram of motion vector orientations shows a significant number of motion vectors clustered about particular orientations, then it may indicate that the corresponding frames of imagery depict human motion. However, if the histogram of motion vector orientations shows a significant number of motion vectors distributed in a relatively uniform manner, then it may indicate that the corresponding frames of imagery depict a boiling event. 
     As another example,  FIG. 8  depicts an example histogram  800  according to an example embodiment of the present disclosure. Histogram  800  describes a distribution of motion vector magnitudes corresponding to the plurality of motion vectors shown on frame  400  of  FIG. 4 , which depicts human motion. In particular, because the motion vectors for frame  400  exhibit relatively large magnitudes, the corresponding histogram  800  reflects a disproportionate number of motion vectors at a particular magnitude having a relatively large value. 
     As another example,  FIG. 9  depicts an example histogram  900  according to an example embodiment of the present disclosure. Histogram  900  describes a distribution of motion vector magnitudes corresponding to the plurality of motion vectors shown on frame  500  of  FIG. 5 , which depicts a boiling event. In particular, because the motion vectors for frame  500  exhibit relatively small magnitudes, the corresponding histogram  900  reflects a disproportionate number of motion vectors at a particular magnitude having a relatively small value. 
     Thus, if an histogram of motion vector orientations shows a significant number of motion vectors clustered about a particular magnitude having a relatively large value, then it may indicate that the corresponding frames of imagery depict human motion. However, if the histogram of motion vector orientations shows a significant number of motion vectors clustered about a particular magnitude having a relatively small value, then it may indicate that the corresponding frames of imagery depict a boiling event. 
     Referring again to  FIG. 2 , at ( 208 ) the one or more histograms generated at ( 206 ) can be respectively compared to one or more pre-learned histograms to determine a dissimilarity score. For example, computing device  104  can implement histogram comparison module  118  to compare the histograms. 
     In particular, the one or more pre-learned histograms stored in memory  108  can respectively describe one or more characteristics of a plurality of motion vectors exhibited by frames of imagery depicting a boiling event. For example, in some embodiments, the pre-learned histograms can be generated through testing or analyzing imagery depicting a boiling event. In other embodiments, the pre-learned histograms can be approximations, such as for example, a histogram having a uniform distribution of orientations. 
     The dissimilarity score can correspond to one or more distance values that respectively result from one or more histogram comparison techniques. For example, the dissimilarity score can be a weighted sum of a plurality of distance values respectively obtained using a plurality of histogram comparison techniques. In some embodiments, a smaller dissimilarity score will be provided for pairs of histograms that exhibit a similar distribution while a larger dissimilarity score will be provided for pairs of histograms that exhibit divergent distributions. 
     At ( 210 ) it can be determined whether the dissimilarity score determined at ( 208 ) is less than a threshold dissimilarity score. 
     If it is determined at ( 210 ) that the dissimilarity score is not less than the threshold dissimilarity score, then method ( 200 ) can proceed to ( 212 ) and clear a counter to zero. After ( 212 ), method ( 200 ) can return to ( 202 ) and obtain an additional frame of imagery depicting the cooktop. In such fashion, consecutive frames can be analyzed in an iterative fashion to identify a boiling event. However, the frames used to determine motion vectors at ( 204 ) are not necessarily consecutive in nature. As an example, one frame can be obtained for analysis each second event even though the vision sensor is collecting frames at a higher rate. 
     Returning to ( 210 ), if it is determined that the dissimilarity score is less than the threshold dissimilarity score, then method ( 200 ) can proceed to ( 214 ) and increment the counter. For example, computing device  104  can increment the counter  112 . 
     At ( 216 ) it can be determined whether the counter value is greater than a threshold counter value. If it is determined at ( 216 ) that the counter value is not greater than the threshold value, then method ( 200 ) can return to ( 202 ) and obtain an additional frame of imagery depicting the cooktop. 
     However, if it is determined at ( 216 ) that the counter value is greater than the threshold value, then method ( 200 ) can proceed to ( 218 ) and provide an alarm. In such fashion, a threshold number of consecutive frames of imagery must be classified as depicting a boiling event prior to providing the alarm. The threshold counter value can be adjustable to adjust for detection confidence. 
     Providing the alarm at ( 218 ) can include flashing a lighting device, sounding a sound with a speaker, or performing various communications methods or protocols to communicate with a mobile device of the user or the cooktop itself. 
     This written description uses examples to disclose the invention, including the best mode, and also to enable any person skilled in the art to practice the invention, including making and using any devices or systems and performing any incorporated methods. The patentable scope of the invention is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they include structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal languages of the claims.