Patent Description:
Liquid detection sensors are capable of detecting the presence of liquid in a medium. Liquid detection sensors typically operate by detecting a change of a property of a portion of the liquid detection sensor that is designed to change in the presence of liquid, such as water. For example, some electronic devices include small stickers that are designed to change color (e.g., from white to red) when the sticker is in contact with water.

Electronic liquid detection sensors conventionally rely on the change in conductivity of a medium to detect the presence of water or other liquids. For example, an electronic liquid detection sensor may include two terminals spaced apart and configured to conduct a current. During normal operation, since there is no electrical conduction path between the two terminals, no current flows between the two terminals during normal operation. When water is present between the two terminals, the water creates an electrical path that allows the flow of current between the two terminals. Therefore, the liquid detection sensor can sense the presence of a liquid between the two terminals when the current flowing through the terminals is greater than a predefined threshold. The electronic liquid detection sensor may, for example, trigger an alarm to alert a user of the presence of water when the current flowing through the two terminals is greater than the predefined threshold.

Conventional electronic liquid detection sensors, therefore, are capable of detecting the presence of liquid when at least a portion of the liquid detection sensor is in contact with the liquid. Publication <CIT> discloses a method for ascertaining and monitoring fill level of a medium in a container by means of a field device with a travel time measuring method. Publication <CIT> pertains to a fill level sensor comprising an antenna device. Publication <CIT> discloses a device for sensing a road before a vehicle and comprising a submillimeter wave sensor sending waves reflected by the road surface. Further radar systems are disclosed in publications <CIT> and<CIT>. Publication <CIT> discloses a robot cleaner comprising a detector able to capture images of regions disposed in front of the robot cleaner.

In accordance with the claimed invention, a device includes: a millimeter-wave radar sensor circuit configured to generate N virtual channels of sensed data, where N is an integer number greater than one; and a processor configured to: generate a 2D radar image of a surface in a field of view of the millimeter-wave radar sensor circuit based on sensed data from the N virtual channels of sensed data, where the 2D radar image includes azimuth and range information, generate a multi-dimensional data structure based on the 2D radar image using a transform function, compare the multi-dimensional data structure with a reference multi-dimensional data structure, and determine whether liquid is present in the field of view of the millimeter-wave radar sensor circuit based on comparing the multi-dimensional data structure with the reference multi-dimensional data structure.

In accordance with the claimed invention, a method for detecting a liquid from a moving vehicle includes: generating a 2D radar image of a surface in a field of view of a millimeter-wave radar sensor circuit based on sensed data from N virtual channels generated by the millimeter-wave radar sensor circuit, wherein N is an integer number greater than one, wherein the 2D radar image comprises azimuth and range information; generating a multi-dimensional data structure based on the 2D radar image using a transform function; comparing the multi-dimensional data structure with a reference multi-dimensional data structure; and determining whether liquid is present in the field of view of the millimeter-wave radar sensor circuit based on comparing the multi-dimensional data structure with the reference multi-dimensional data structure.

In accordance with an embodiment, a vacuum cleaner robot includes: a millimeter-wave radar sensor circuit configured to generate N virtual channels of sensed data, where N is an integer number greater than one; and a processor configured to: generate a 2D radar image of a surface in a field of view of the millimeter-wave radar sensor circuit based on sensed data from the N virtual channels of sensed data, where the 2D radar image includes azimuth and range information, generate a multi-dimensional data structure based on the 2D radar image using a transform function, compare the multi-dimensional data structure with a reference multi-dimensional data structure, and determine whether liquid is present in the field of view of the millimeter-wave radar sensor circuit based on comparing the multi-dimensional data structure with the reference multi-dimensional data structure.

To more clearly illustrate certain embodiments, a letter indicating variations of the same structure, material, or process step may follow a figure number.

The making and using of the presently preferred embodiments are discussed in detail below.

The description below illustrates the various specific details to provide an in-depth understanding of several example embodiments according to the description. In other cases, known structures, materials or operations are not shown or described in detail so as not to obscure the different aspects of the embodiments. References to "an embodiment" in this description indicate that a particular configuration, structure or feature described in relation to the embodiment is included in at least one embodiment. Consequently, phrases such as "in one embodiment" that may appear at different points of the present description do not necessarily refer exactly to the same embodiment. Furthermore, specific formations, structures or features may be combined in any appropriate manner in one or more embodiments.

The present invention will be described with respect to embodiments in a specific context, a system and method for detecting liquids, such as water, by using a millimeter-wave radar. Using a millimeter-wave radar for liquid detection allows for the detection of liquids in applications that are not in contact with the liquid. In one embodiment, an array of radar sensors is used to form a two-dimensional image of a surface that is moving with respect to the radar sensor. Embodiments can be directed toward such applications as robotic vacuum cleaners, conveyer belt monitoring systems, and other types of systems that are sensitive to the presence of liquid. Advantages of such embodiments include the ability to detect liquids in unknown surfaces. Additional advantages include the detection of liquids in surfaces that are moving with respect to the millimeter-wave radar at various speeds.

In an embodiment of the present invention, a millimeter-wave radar determines whether liquid is present in a field of view of the millimeter-wave radar by generating a 2D image and comparing the 2D image with one or more reference images. The 2D image includes azimuth and depth information of the field of view. A normalization step is performed to compensate for velocity and vibration motions of the vacuum cleaner robot before comparing the 2D image with the one or more references. The one or more reference images are generated using artificial intelligence (AI) algorithms.

In an embodiment, a millimeter-wave radar may be used to detect moving and static objects in the field of view of the millimeter-wave radar, and distinguish liquids from other objects in the field of view. For example, <FIG> shows radar system <NUM>, according an embodiment of the present invention. Radar system <NUM> includes millimeter-wave radar <NUM> and processor <NUM>.

During normal operation, millimeter-wave radar <NUM> transmits a plurality of radiation pulses <NUM>, such as chirps, to scene <NUM>. The transmitted radiation pulses <NUM> are reflected in objects of scene <NUM>. The reflected radiation pulses (not shown in <FIG>), which are also referred to as the echo signal, are detected by millimeter-wave radar <NUM> and processed by processor <NUM> to, for example, detect liquids.

The objects in scene <NUM> may include liquids <NUM>, such as water, moving objects <NUM> and static objects <NUM>. Other objects may also be present in scene <NUM>.

Processor <NUM> analyses the echo data using signal processing block <NUM> to identify objects in the field of view of millimeter-wave radar <NUM>. For example, signal processing block <NUM> may use a range Fast Fourier Transform (FFT) to identify range bins in which objects are located.

Processor <NUM> uses determination block <NUM> to determine whether any of the identified objects in the field of view of millimeter-wave radar <NUM> is a liquid. For example, in some embodiments, determination block <NUM> compares the processed echo data with one or more signatures (or a database of signatures) to determine whether the identified objects is a liquid. Processor <NUM> may generate a data signal that indicates whether a liquid has been detected in the field of view of millimeter-wave radar <NUM>.

Processor <NUM> may be implemented as a general purpose processor, controller or digital signal processor (DSP). In some embodiments, processor <NUM> may be implemented as a custom application specific integrated circuit (ASIC). In some embodiments, processor <NUM> includes a plurality of processors, each having one or more processing cores. Alternatively, each embodiment function may be implemented using dedicated logic. In other embodiments, processor <NUM> includes a single processor having one or more processing cores.

Millimeter-wave radar <NUM> includes a millimeter-wave radar sensor circuit and one or more antennas (not shows). For example, the millimeter-wave radar sensor circuit may be implemented using a two-dimensional millimeter-wave phase-array radar that transmits and receives signals in the <NUM> to <NUM> range. Alternatively, frequencies outside of this range may also be used. Some embodiments may include a single millimeter-wave radar sensor circuit. Other embodiments use a plurality of millimeter-wave radar sensor circuits, the data of which are gathered and processed by processor <NUM>, which may be implemented as a centralized processing device.

In some embodiments, millimeter-wave radar <NUM> includes a uniform linear array antenna. The echo signals received are filtered and amplified using band-pass filter (BPFs), low-pass filter (LPFs), mixers, low-noise amplifier (LNAs), and intermediate frequency (IF) amplifiers in ways known in the art. The echo signals are then digitized using one or more analog-to-digital converters (ADCs) for further processing. Other implementations are also possible.

Identifying liquids without direct physical contact is advantageous in various applications. For example, <FIG> shows a diagram of vacuum cleaner robot <NUM>, according to an embodiment of the present invention. Vacuum cleaner robot <NUM> includes two millimeter-wave radars <NUM> disposed in the front of vacuum cleaner robot <NUM>. Each millimeter-wave radar <NUM> has field of view <NUM> in a direction towards forward movement of vacuum cleaner robot <NUM>.

The top portion of <FIG> shows a top view of vacuum cleaner robot <NUM> moving towards liquid <NUM> in floor <NUM>. The bottom portion of <FIG> shows a side view of vacuum cleaner robot <NUM> over floor <NUM>. As shown in <FIG>, each of field of view <NUM> begins at height h<NUM>, has field of view azimuth angle α, field of view elevation angle β and covers a forward distance d<NUM>.

During normal operation, vacuum cleaner robot <NUM> moves in the x direction with velocity v<NUM>. Millimeter-wave radars <NUM> continuously transmit radiation pulses, receive the echo, process the echo data and determine whether liquid is present in field of view <NUM>. When liquid is detected in field of view <NUM>, vacuum cleaner robot <NUM> takes a predetermined action. In some embodiments, the predetermined action is taken regardless of the location of the detected liquid in field of view <NUM>. Examples of predetermined actions are: decrease the velocity of movement, stop, move in a different direction (e.g., left, right, or reverse), make a sound, turn on a light, or a combination thereof. Other actions may also be taken.

Field of view <NUM> covers distance d<NUM> (e.g., <NUM>), has field of view azimuth angle α, which may be, for example <NUM>°, and elevation angle β, which may be, for example, <NUM>°. In some embodiments, field of view azimuth angle α may be <NUM> to 75deg. In some embodiments, field of view elevation angle β may be higher than <NUM>°, such as <NUM>° or higher, or may be lower than <NUM>°, such as <NUM>° or lower. In some embodiments, angle β is between <NUM>° and <NUM>°. In some embodiments, distance d<NUM> may be higher than <NUM>, such as <NUM>, <NUM>, or higher, or may be lower than <NUM>, such as <NUM>, <NUM>, or lower.

As shown, vacuum cleaner robot <NUM> includes two millimeter-wave radars <NUM> as a specific example. It should be appreciated, however, that embodiments may include a single millimeter-wave radar <NUM> while other embodiments may include more than two millimeter-wave radars <NUM>. In some embodiments, vacuum cleaner robot <NUM> includes a plurality of identical millimeter-wave radars. Other embodiments may include different millimeter-wave radars.

Vacuum cleaner robot <NUM> includes millimeter-wave radars <NUM> disposed in the front of vacuum cleaner robot <NUM> at height h<NUM> (e.g., <NUM>) and with a field of view directed to the front of vacuum cleaner robot <NUM>. In some embodiments, at least one of millimeter-wave radars <NUM> may be disposed in other portions of vacuum cleaner robot <NUM> and at different heights. For example, in some embodiments, one of millimeter-wave radars <NUM> may be disposed in the back vacuum cleaner robot <NUM> and with a field of view towards the back of vacuum cleaner robot <NUM>. Such positioning is advantageous in case vacuum cleaner robot <NUM> is capable of moving in reverse. Other embodiments may position at least one millimeter-wave radar <NUM> in a side of vacuum cleaner robot <NUM> and with a field of view towards the side of vacuum cleaner robot <NUM>. Such positioning is advantageous in case vacuum cleaner robot <NUM> turns and moves toward the side direction.

In the illustrated example, liquid <NUM> is static and has a diameter d<NUM>, such as <NUM>. It should be understood that liquid <NUM> may have different dimensions from that which is illustrated. For example, liquid <NUM> may have a larger diameter, such as <NUM> or higher, or smaller diameter, such as <NUM> or smaller. The shape of liquid <NUM> may be symmetric, asymmetric, and may have various heights depending on the specific scenario. In some embodiments, liquid <NUM> may not be static.

Liquid <NUM> may be, for example, water or a water based liquid, urine, such as cat or dog urine, or other types of liquids. For example, liquid <NUM> may be a detergent or other chemical.

Vacuum cleaner robot <NUM> may operate on various types of surface, and travel at various speeds. Although millimeter-wave radars <NUM> are not moving with respect to vacuum cleaner robot <NUM>, and some of the objects in field of view <NUM> may be static with respect to floor <NUM>, as vacuum cleaner robot <NUM> moves, objects may appear as moving objects to millimeter-wave radars <NUM>. Additionally, the vibration of vacuum cleaner robot <NUM> as vacuum cleaner robot moves across floor <NUM> may appear as vibrations of objects in the field of view to millimeter-wave radars <NUM>.

<FIG> shows embodiment method <NUM> for generating 2D image <NUM> for detecting a liquid with millimeter-wave radar <NUM> as used in vacuum cleaner robot <NUM>, according to an embodiment of the present invention. Method <NUM> may be performed, for example, by processor <NUM>.

During step <NUM>, which includes steps <NUM> and <NUM>, radiation pulses are transmitted by a radar system, reflected by objects within the field of view of the radar system, and received by a radar system, such as millimeter-wave radar <NUM>. During step <NUM>, a series of radiation pulses, such as chirps, are transmitted toward a scene, such as scene <NUM>. Some embodiment radar systems may transmit, for example, <NUM> chirps during a <NUM> period. Alternatively, a different number of chirps (e.g., <NUM>-<NUM> chirps) over a different period (e.g., <NUM> or <NUM>) may be transmitted. In some embodiments, the number of chirps transmitted over a predefined period is a power of <NUM>.

The echo signals are received during step <NUM> after the radiation pulses are reflected into objects from the scene. The radiation pulses are transmitted from two transmitter elements TX1 and TX2 and are received by two receiver elements RX1 and RX2. For example, at a first time, transmitter element TX1 transmits <NUM> chirps, which are reflected over objects and received by receiver elements RX1 and RX2, creating virtual channels <NUM> and <NUM>. At a second time, transmitter element TX2 transmits <NUM> chirps, which are reflected over objects and received by receiver elements RX1 and RX2, creating virtual channels <NUM> and <NUM>. Some embodiments may use more than two transmitter elements and/or more than two receiver elements.

Each of the virtual channels <NUM>, <NUM>, <NUM> and <NUM>, generates respective range vectors <NUM>, <NUM>, <NUM>, and <NUM> during step <NUM>. Each of range vectors <NUM>, <NUM>, <NUM> and <NUM> has L range bins, such as <NUM>. Some embodiments may have less (e.g., <NUM>), or more (e.g., <NUM>, <NUM>, or more) range bins in each range vector.

Since millimeter-wave radar <NUM> is moving at the same velocity and with the same vibration as vacuum cleaner robot <NUM>, the same velocity and vibration information is present in all virtual channels <NUM>, <NUM>, <NUM>, and <NUM>. During step <NUM>, the velocity and vibration information is extracted from range vector <NUM> and is used to cancel the velocity and vibration information from virtual channels <NUM>, <NUM>, <NUM>, and <NUM> to generated normalized range vectors <NUM>, <NUM>, <NUM>, and <NUM>. In some embodiments, a different virtual channel may be used as the reference for performing the normalization step.

In some embodiments, the radial velocity/Doppler component of the target modulates the phase of the received signal at a given range bin (corresponding to its distance) along consecutive pulses at a fixed pulse repetition time. If the kth target scatterer introduces ωk Doppler then the vectorized signal along the slow time can be expressed as <MAT> where ρk accounts for constants along this dimension, the time index ts indicates slow time, TPRT represents Pulse Repetition Time and NP represents the number of pulses in a frame/dwell.

Some embodiments maximize the output signal-to-noise ratio (SNR) in the receiver processing to improve Doppler detection. Maximizing the output SNR may be achieved by matched filtering, which is a time-reversed, conjugate version of the signal. In some embodiments, the matched filter is given by <MAT>.

Hence Discrete-Time Fourier Transform (DTFT) is max-SNR detector for estimating target's radial velocity/Doppler/vibration and thus coherent processing technique involves deploying FFT along the slow time.

During step <NUM>, a beamspace transformation is performed, in which each of the range bins of normalized range vectors <NUM>, <NUM>, <NUM>, and <NUM> is expanded to have N bins, where N may be, for example, <NUM>. In some embodiments, N may be <NUM>, <NUM>, <NUM>, or another number.

A 2D image <NUM> is generated with L range bins and N azimuth bins from normalized range vectors <NUM>, <NUM>, <NUM> and <NUM>. Since 2D image <NUM> is generated from normalized range vectors, 2D image <NUM> is independent, or loosely dependent from the vibration and velocity of vacuum cleaner robot <NUM>. The normalization step, therefore, advantageously facilitates the comparison step with one or more reference 2D images to determine the presence of liquids in the field of view of the millimeter-wave radar. The normalization step allows for detection of liquids when vacuum cleaner robot <NUM> is moving as well as when vacuum cleaner robot <NUM> is not moving, or moving slowly.

<FIG> shows example 2D images, according to an embodiment of the present invention. The top portion of <FIG> shows 2D image <NUM> illustrating that no liquid is present in the field of view of the millimeter-radar. The bottom portion of <FIG> shows 2D image <NUM> illustrating that liquid is present from nearest detectable distance (range bin o) up to distance d<NUM> (range bin <NUM>) at the center of the azimuth range (bin <NUM> of <NUM>).

In some embodiments, the beamspace transformation is performed in all range bins to generate 2D image <NUM>. In other embodiments, only the range bins with identified objects are beamspace transformed. The range bins without identified objects are populated with, e.g., zeros, in 2D image <NUM>. By performing the beamspace transformation on only the range bins with identified objects, the computation power is reduced, and the speed of generation of 2D image <NUM> is increased. Increasing the speed of generation of 2D image <NUM> is advantageous to allow enough time for vacuum cleaner robot <NUM> to react to the presence of liquid in the field of view of millimeter-wave radar <NUM>.

<FIG> illustrates a block diagram of embodiment method <NUM> of performing a millimeter-wave sensor based liquid detection. Method <NUM> may be implemented by, e.g., vacuum cleaner robot <NUM>. Radar processing occurs as follows. In steps <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, radar data is collected from the millimeter-wave radar sensor and objects are detected in the field of view of the millimeter-wave radar sensor. In step <NUM>, and <NUM>, a range-cross-range 2D image having azimuth and depth information is generated, in part, using a Capon/MVDR analysis. During steps <NUM>, <NUM> and <NUM>, the 2D image is transformed according to a predictive model and is compared with 2D reference images of a signature database to determine whether liquid is present in the field of view.

In step <NUM>, live radar data is collected from the millimeter wave radar sensor. In some embodiments, this radar data is collected form digitized baseband radar data and may include separate baseband radar data from multiple antennas. In some embodiments, these antennas may be "virtual antennas" as explained above.

In step <NUM>, signal conditioning, low pass filtering and background removal is performed. During step <NUM>, radar data received during step <NUM> is filtered, DC components are removed, and IF data is filtered to, e.g., remove the Tx-Rx self-interference and optionally pre-filtering the interference colored noise. In some embodiments, filtering includes removing data outliers that have significantly different values from other neighboring range-gate measurements. Thus, this filtering also serves to remove background noise from the radar data. In a specific example, a Hampel filter is applied with a sliding window at each range-gate to remove such outliers. Alternatively, other filtering for range preprocessing known in the art may be used.

In step <NUM>, a series of FFTs are performed on conditioned radar data produced by step <NUM>. In some embodiments, a windowed FFT having a length of the chirp (e.g., <NUM> samples) is calculated along each waveform for each of a predetermined number of chirps in a frame of data. Alternatively, other frame lengths may be used. The FFTs of each waveform or chirp may be referred to as a "range FFT. " In alternative embodiments, other transform types could be used besides an FFT, such as a Discrete Fourier Transform (DFT) or a z-transform. In step <NUM>, the results of each range FFT are stored in slow time.

In step <NUM>, a Doppler FFT is derived based on a series of range FFTs collected in slow time. In some embodiments, calculating the Doppler FFT entails calculating a windowed two-dimensional FFT of the range FFT over slow-time to determine the velocity and vibration of detected objects. Since such velocity and vibration relates to the velocity and vibration of the vacuum cleaner robot, the velocity and vibration information can be used to remove the velocity and vibration components from the range FFT data, as explained with respect to <FIG>.

In various embodiments, a beam is formed at the transmitter by post processing a plurality of baseband signals based on a plurality of signals received by different receivers or a combination thereof. Implementing beamforming by post processing received baseband signals may allow for the implementation of a low complexity transmitter.

In one example, a millimeter-wave sensor system is used with Nt = <NUM> transmit (TX) elements and Nr = <NUM> receive (RX) elements arranged in a linear array. Accordingly, there are Nt × Nr = <NUM> distinct propagation channels from the TX array to the RX array in a linear array configuration for azimuth angle profiling. If the transmitting source (TX channel) of the received signals can be identified at the RX array, a virtual phased array of Nt × Nr elements can be synthesized with Nt + Nr antenna elements. In various embodiments, a time division multiplexed MIMO array provides a low cost solution to a fully populated antenna aperture capable of near field imaging.

In some embodiments, a symmetrical linear arrangement of the TX and the RX elements with some vertical offset between the TX array and the RX array for reduced coupling may be used. For example, with respect to <FIG>, the TX and RX elements of millimeter-wave radar <NUM> may be disposed with the arrangement shown in <FIG>, where the TX elements illumination and field of view direction is directed towards the front of vacuum cleaner robot <NUM>.

<FIG> shows a coordinate axes used to illustrate an embodiment algorithm. As shown, <FIG> illustrates the position of target <NUM>, transmit (TX) antenna element <NUM> and receive (RX) antenna element <NUM>. The position r of target <NUM> can be represented as <MAT> where R is the distance from the origin to target <NUM>. The directional vector u of target <NUM> can be expressed as <MAT>.

Denoting the 3D positional coordinates of the TX antenna element as <MAT>, i = <NUM>, <NUM> and the RX antenna element as <MAT>, j = <NUM>, <NUM> in space, then on assuming far field conditions, the signal propagation from a TX element <MAT> to target <NUM> (assumed to be a point scatterer) and subsequently the reflection from target <NUM> to Rx antenna element <MAT> can be approximated as <NUM> * x + dij, where x is the based distance of target <NUM> to the center of the virtual linear array, and dij refers to the position of the virtual element to the center of the array.

The transmit steering vector may be written as: <MAT> and the receiving steering vector may be expressed as: <MAT> where λ is the wavelength of the transmit signal. A joint TX and RX steering vector a(θ, ϕ) can be derived as the Kronecker of the transmit and receive steering vectors (assuming i = j = <NUM>): <MAT> From the joint steering vector, the following beamspace spectrum may be computed from which angles θ and ϕ may be estimated according to a minimum variance distortionless response (MVDR) algorithm: <MAT> In the above expression, C = E{x(r, d)x(r, d)H} is calculated as a covariance matrix, where E{. } is the expectation operator. The above covariance matrix may be estimated as sample matrix indicator as <MAT> where xi(r, d) represents measured range, Doppler data (r, d).

For the generation of 2D images in which azimuth and range are considered, the value of angle ϕ may be known or assumed and the determination of angle ϕ may be omitted. For example, in some embodiments, ϕ is equal to zero. In various embodiments, a MVDR algorithm is applied as follows.

In step <NUM> data is saved from all virtual antennas in a line of detected range-Doppler bins. In step <NUM>, the antenna covariance matrix of the detected range-Doppler bins is estimated as follows: <MAT> where Rr,d is antenna covariance matrix, xr,d(n) represents the data over a particular (range, Doppler) = (r,d) and n represents the specific (r,d) data across multiple frames (n being the indices, and N is the number of frames considered). In step <NUM>, a MVDR algorithm is applied to the range and Doppler data as follows using the above derived covariance matrix: <MAT> where P(θ) represents azimuth spatial spectrum and a(θ) is the virtual antenna steering vector along the azimuth angle for test angle θ within the field of view. In an embodiment, the value θ is found that provides a peak value for P(θ). This determined value for θ is the estimated azimuth angle θest of the detected foreign object.

In step <NUM>, a range-cross-range 2D image having azimuth and range information is generated. In some embodiments, the 2D image includes information for all range bins. In other embodiments, the 2D image only includes information in the range bins in which objects have been identified. Range bins without an identified object are populated with, e.g., zeros.

In step <NUM>, also referred to as transformation step or embedding step, the 2D image is transformed using a prediction model generated by an embedding process. During the transformation step, the 2D image is mapped into a transformed 2D image or vector that allows for easy liquid identification. For example, although a liquid object and a non-liquid object may be close to each other in the 2D image, the liquid object and the non-liquid object are far from each other (in Euclidean terms) in the transformed 2D image or vector. The transformed 2D image is compared with one or more reference signatures of a signature database using a nearest neighbor algorithm to determine whether a liquid is present in the field of view.

The application implementing method <NUM>, such as a vacuum cleaner robot, may take an action based on whether liquid is detected. For example, when liquid is detected. The vacuum cleaner robot may decrease the velocity of movement, stop, move in a different direction (e.g., left, right, or reverse), make a sound, turn on a light, or a combination thereof. Other actions may also be taken.

Vacuum cleaner robot detects liquids of various diameters in various types of floors and while moving at various speeds or when not moving. <FIG> illustrates a block diagram showing a machine learning pipeline for machine language based feature extraction and identification that can be used to generate reference signatures (step <NUM>) to classify an object as liquid (step <NUM>). The top portion <NUM> of <FIG> is devoted to the processing storage of features for comparison to later measurements. The data and steps shown in this portion represent the actions performed when radar measurements are performed and processed for a classification category. The bottom portion <NUM> is devoted to the processing and comparison of new measurements for comparison to stored data. These data and steps represent the actions performed when the system is identifying and detecting liquids.

As shown in the top portion <NUM> of <FIG>, training data <NUM> is transformed into stored feature vectors <NUM> and corresponding labels <NUM>. Training data <NUM> represents the raw data (e.g., echo). Feature vectors <NUM> represent sets of generated vectors that are representative of the training data <NUM>. Labels <NUM> represent user metadata associated with the corresponding training data <NUM> and feature vectors <NUM>.

As shown, training data <NUM> is transformed into feature vectors <NUM> using embodiment image formation algorithms. Data preparation block <NUM> represents the initial formatting of raw sensor data, and data annotation block <NUM> represents the status identification from training data <NUM>.

During operation, one or more radar images are taken of a controlled environment that includes one or more liquid and/or non-liquid objects using millimeter-wave sensors described above. In some cases, multiple radar images are recorded to increase the accuracy of identification. Embedding deep neural network <NUM> evaluates the ability of an embedding model <NUM> to identify feature vectors and iteratively updates training data <NUM> to increase the classification accuracy of the algorithm. The training performance of the machine learning algorithm may be determined by calculating the cross-entropy performance. In some embodiments, the embedding deep neural network514 iteratively adjusts image formation parameters for a classification accuracy of at least <NUM>%. Alternatively, other classification accuracies could be used.

Embedding deep neural network <NUM> may be implemented using a variety of machine learning algorithms known in the art. For example, a neural network algorithms, such as comma. ai, Nvidia SDC CNN, LeCunn Net, or other neural network algorithms known in the art, may be used for classification and analysis of stored feature vectors <NUM>. During the iterative optimization of stored feature vectors <NUM>, a number of parameters of image formation <NUM> may be updated.

Once the system has been trained using reference training data <NUM>, the reference signatures may be used for classification during normal operation. During normal operation, new target data <NUM> is received. Data preparation block <NUM> prepares the new target data <NUM> for image formation, and image formation block <NUM> forms new extracted feature vectors <NUM>. Embedding model <NUM> utilizes embedding deep neural network <NUM> to match new extracted feature vectors <NUM> to a stored feature vector <NUM>. When a match is identified, a predicted label is provided that identifies the new feature vector. In some embodiments, data from the stored labels <NUM> is provided as a predicted label. Embedding model <NUM> may be a machine learning model with optimal parameters computed/evaluated through a machine learning algorithm.

The normalization step simplifies the reference signature generation during the training phase by removing speed of movement and associated vibration as a variable. In other words, the training phase, as described with respect to <FIG>, may be performed with static images rather than generating images with vacuum cleaner robot moving at different speeds. During the identification phase, the same normalization step is applied to the 2D image before it is compared with the reference signatures to identify liquids in the field of view.

To facilitate identification of liquids over various types of floors having different surfaces and materials, an embedding process is used to generate embedding vector in higher dimensional transformed space (steps <NUM>, <NUM>). <FIG> illustrates a block diagram showing an embedding process flow, according to an embodiment of the present invention. The left portion <NUM> of <FIG> is devoted to the embedding process during the training phase. The right portion <NUM> is devoted to an inference phase, in which processing and comparison of new images using the embedded prediction model with reference images takes place.

During the training phase, images from set of images <NUM> are analyzed three at a time. For example, during step <NUM>, a deep neural network module receives a first 2D image of a first floor with a liquid, a second 2D image of a second floor with a liquid, and a third 2D image of a floor without a liquid. The deep neural network module generates respective vectors for the first, second and third 2D images, which are stored in an example database during step <NUM>. The deep neural network module then modifies the measurement/transform function so that the generated vectors associated with having liquids (e.g., first and second 2D images in this example) are close to each other (in Euclidean terms) and are far from the vectors associated with not having liquids (e.g., third 2D image in this example) in Euclidean terms. In this example, the embedding process modifies the measurement/transform function such that the first and second vectors are closer to each other than the second and third vectors.

Steps <NUM> and <NUM> are repeated for all 2D images of set <NUM>. In some embodiments, different permutations of 2D images from set <NUM> are analyzed during steps <NUM> and <NUM>. Set <NUM> may have thousands or tens of thousands of 2D images.

After iterating through steps <NUM> and <NUM>, the resulting deep neural network module measurement/transform function is tuned to determine whether liquid is present in various types of floors. The resulting deep neural network module is also referred to as the prediction deep neural network model, which is used during the inference phase. Since the measurement/transform function has been tuned to distinguish floors with liquids from floors without liquids, accurate detection of liquids in floors that were not used during the training phase is possible.

In some embodiments, the vectors generated by the deep neural network module are vectors in a, e.g., <NUM> or <NUM> dimensional space. Some embodiments may use a different number of dimensions. In some embodiments, the deep neural network module may generate other multi-dimensional data structures instead of vectors, such as, matrices. In other embodiments, deep neural network module may generate vectors in other dimensional spaces.

During the inference phase (e.g., during normal operation), new 2D image <NUM> is received. New 2D image <NUM> may be generated as described with respect to methods <NUM> and <NUM>. The embedding deep neural network model <NUM> generates a new vector using the measurement/transform function generated during the training phase. During step <NUM>, the new vector is compared with the database of reference vectors generated during step <NUM> to determine whether liquid is detected in new 2D image <NUM>. The k nearest neighbor algorithm may be used to compare the new vector with the reference vectors during step <NUM>.

In some embodiments, millimeter-wave radar <NUM> includes a transparent enclosure (i.e., transparent to the transmitted and received frequencies used by millimeter-wave radar <NUM>) that at least partially encloses the TX and RX elements of millimeter-wave radar <NUM>. Due to the material and geometric properties of the transparent enclosure, some of the pulses transmitted by the TX elements may be reflected by the transparent enclosure instead of by objects in the field of <NUM>. Reflections from the transparent enclosure may create backscatter that appears as noise in the echo data and 2D images analyzed in methods <NUM> and <NUM>. High amounts of backscatter may result in improperly determining whether liquid is present in the floor.

<FIG> shows millimeter-wave radar <NUM> having transparent enclosure <NUM>, according to an embodiment of the present invention. As shown in <FIG> (not to scale), millimeter-wave radar <NUM> includes transparent enclosure <NUM>. Field of view <NUM> covers the area between field of view lines <NUM> and <NUM>, and has centerline <NUM> that is orthogonal to surface axis <NUM>, and axis <NUM> is parallel to a vertical side wall of transparent enclosure <NUM>.

Backscatter is minimized when angle <NUM> between axis <NUM> and axis <NUM> is between <NUM>° and <NUM>°. For example, in some embodiments, angle <NUM> is <NUM>°. A different angle β may be achieved while minimizing backscatter by rotating transparent enclosure <NUM> while keeping angle <NUM> fixed.

Transparent enclosure <NUM> may be implemented with plastic, glass, or other types of materials. For example, some embodiments may implemented transparent enclosure <NUM> with Polycarbonate, Polyamide or ABS.

In some embodiments, angle β is selected to minimize the blind spot immediate to the radar platform while maximizing the range of the field of view and minimizing backscatter.

<FIG> shows additional field of view details of millimeter-wave radar <NUM> as shown in <FIG>, according to an embodiment of the present invention. Transparent enclosure <NUM> has been omitted in <FIG> for clarity purposes. In an embodiment, angle <NUM> is <NUM>°, angle <NUM> is <NUM>°, angle <NUM> is <NUM>°, angle <NUM> is <NUM>°, height h<NUM> is <NUM>, blind distance d<NUM> is <NUM>, and distance d<NUM> is <NUM>. In other embodiments, angle <NUM> is <NUM>°, angle <NUM> is <NUM>°, angle <NUM> is <NUM>°, angle <NUM> is <NUM>°, height h<NUM> is <NUM>, blind distance d<NUM> is <NUM>, and distance d<NUM> is <NUM> and distance d<NUM> is <NUM>. Other values for angles <NUM>, <NUM>, <NUM>, <NUM>, distances d<NUM>, d<NUM>, and d<NUM>, and height h<NUM> are possible.

<FIG> shows a top view of millimeter-wave radar <NUM>, as shown in <FIG>, according to an embodiment of the present invention. Transparent enclosure <NUM> has been omitted in <FIG> for clarity purposes. In an embodiment, angle α is <NUM>°, angle <NUM> is <NUM>°, distance ds is <NUM>, distance d<NUM> is <NUM>, distance d<NUM> is <NUM>, distance d<NUM> is <NUM>, where distance d<NUM> is the distance between edges of vacuum cleaner robot <NUM>. Other values for angles α and <NUM>, and distances d<NUM>, d<NUM>, d<NUM>, and d<NUM> are possible.

Various applications, other than vacuum cleaner robot <NUM>, may implement the embodiments disclosed. For example, other mobile applications, such as other mobile robots or vehicles having a millimeter-wave radar attached may implement the embodiments disclosed.

Liquid detection may also be implemented in applications in which the millimeter-wave radar is static and the surface containing the liquid is moving. For example, <FIG> shows a diagram of conveyor belt system <NUM>, according to an embodiment of the present invention. Conveyor belt system <NUM> may be, for example, a conveyor belt system in a warehouse, factory, airport security, or supermarket. Conveyor belt system <NUM> includes conveyor belt <NUM> and millimeter-wave radar <NUM> that has field of view <NUM> in a direction towards belt surface <NUM>. Millimeter-wave radar <NUM> operates in a similar manner in conveyor belt system <NUM> than in vacuum cleaner robot <NUM>. In conveyor belt system <NUM>, however, instead of millimeter-wave radar <NUM> moving with respect to surface <NUM>, belt surface <NUM> moves with respect to millimeter-wave radar <NUM>. Conveyor belt system may implement methods <NUM>, <NUM>, <NUM>, and <NUM>, for detecting a liquid in a surface.

The top portion of <FIG> shows a side view of conveyor belt system <NUM> where belt surface moves objects towards the left of <FIG> (e.g., moving liquid <NUM> towards field of view <NUM>). The bottom portion of <FIG> shows a top view of conveyor belt system <NUM>. As shown in <FIG>, field of view <NUM> begins at height h<NUM> with respect to belt surface <NUM>, has field of view azimuth angle α, field of view elevation angle β and covers a forward distance d<NUM>.

During normal operation, conveyer belt surface <NUM> moves in the x direction with velocity v<NUM>. Millimeter-wave radars <NUM> operates in a similar manner as described with respect to <FIG>. When liquid is detected in field of view <NUM>, conveyor belt system <NUM> takes a predetermined action. Examples of predetermined actions are: decrease the velocity of movement of belt surface <NUM>, stop movement of belt surface <NUM>, move belt surface <NUM> in the reverse direction, make a sound, turn on a light, or a combination thereof. Other actions may also be taken.

Field of view <NUM> covers distance d<NUM> (e.g., <NUM>), has field of view azimuth angle α, which may be, for example <NUM>°, and elevation angle β, which may be, for example, <NUM>°. In some embodiments, field of view azimuth angle α may be higher than <NUM>°, such as <NUM>° or higher, or may be lower than <NUM>°, such as <NUM>° or lower. In some embodiments, field of view azimuth angle α may be between <NUM>° and <NUM>°. In some embodiments, field of view elevation angle β may be higher than <NUM>°, such as <NUM>° or higher, or may be lower than <NUM>°, such as <NUM>° or lower. In some embodiments, angle β is between <NUM>° and <NUM>°. In some embodiments, distance d<NUM> may be higher than <NUM>, such as <NUM>, <NUM>, or higher, or may be lower than <NUM>, such as <NUM>, <NUM>, or lower. In some embodiments, angle β is selected based on height h<NUM> to optimize field of view <NUM>, as described, for example, with respect to <FIG>.

Conveyor belt system <NUM> includes a single millimeter-wave radar <NUM>. Some embodiments may include more than one millimeter-wave radar <NUM>, which may or may not be identical to each other.

Claim 1:
A device (<NUM>) comprising:
a millimeter-wave radar sensor circuit (<NUM>) configured to generate N virtual channels (<NUM>, <NUM>, <NUM>, <NUM>) of sensed data, wherein N is an integer number greater than one; and
a processor (<NUM>) configured to:
generate a 2D radar image (<NUM>) of a surface in a field of view (<NUM>) of the millimeter-wave radar sensor circuit (<NUM>) based on sensed data from the N virtual channels (<NUM>, <NUM>, <NUM>, <NUM>) of sensed data, wherein the 2D radar image (<NUM>) comprises azimuth and range information,
generate a multi-dimensional data structure based on the 2D radar image (<NUM>) using a transform function,
compare the multi-dimensional data structure with a reference multi-dimensional data structure, and
determine whether liquid (<NUM>) is present in the field of view (<NUM>) of the millimeter-wave radar sensor circuit (<NUM>) based on comparing the multi-dimensional data structure with the reference multi-dimensional data structure.