Patent Description:
In the US alone there is an annual average of <NUM> fatal drowning in residential swimming pools (source: CDC). <NUM>% of death among ages <NUM>-<NUM> is caused by drowning, while at those ages <NUM>% of fatal incidents and <NUM>% of nonfatal incidents occur in swimming pools. Over <NUM> are treated in emergency rooms, annually, as a results of drowning, or near-drowning events.

Safety means that aim to reduce such accidents include fences, pool covers, alarms based on floats, wearable devices, and video analytics based systems. Fences, pool covers, and float-based alarms prevent access to the water, while armed, but are not active once disarmed, and do not work while the pool is in use. Therefore, relying on those means alone means that during standard pool use, the only safety mean is adult supervision, which can be sometimes distracted. Moreover, float-based alarms are sensitive to any water movement and therefore cannot be activated while the pool cleaner robot is in use. Wearable devices include floaties for infants and young children, and wearable bracelets or necklaces for bigger kids. All those require the responsible adults to force and supervise wearing the items, and require to have enough units for all the kids at all times.

There are several video analytic based products/methods that aim to prevent drowning. Some use only above water cameras, and either aim to detect entrance to the pool or to detect early stress signs. These systems suffer from the difficulty to distinguish between stress signs and normal playing movements. Examples of such a method are described in "<NPL>, and "<NPL>).

There are systems that use underwater cameras as well, and intend to detect near drowning events by recognizing lack of movement of a submerged person. However, they are based on a technology that requires that each point inside the pool be viewed by at least two cameras. Data from multiple cameras is used when detecting objects/people, tracking them, and for deciding whether they are in a drowning risk. These systems were designed for commercial pools, to be used as an assisting tool for lifeguards. These systems need to cover a very large body of water and therefore use several overhead and underwater cameras. Document <CIT> (<NUM>-<NUM>-<NUM>) discloses such a system.

These systems are very costly and require complicated installation.

In accordance with certain aspects of the presently disclosed subject matter, there is provided a method according to claim <NUM>.

In accordance with certain aspects of the presently disclosed subject matter, there is provided a system according to claim <NUM>.

According to some embodiments, the solution proposes an affordable device, which allows a simple installation and does not require the presence of an expert.

According to some embodiments, no complicated calibration is needed to install the device.

According to some embodiments, a solution which is designed especially for relatively small pools is described.

These embodiments can be combined according to any of their possible technical combination.

In order to understand the invention and to see how it can be carried out in practice, embodiments will be described, by way of non-limiting examples, with reference to the accompanying drawings, in which:.

However, it will be understood by those skilled in the art that the presently disclosed subject matter may be practiced without these specific details. In other instances, well-known methods have not been described in detail so as not to obscure the presently disclosed subject matter.

Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as "detecting", "tracking", "associating", "clustering",or the like, refer to the action(s) and/or process(es) of a processor that manipulate and/or transform data into other data, said data represented as physical, such as electronic, quantities and/or said data representing the physical objects.

The term "processing unit" covers any computing unit or electronic unit that may perform tasks based on instructions stored in a memory, such as a computer, a server, a chip, etc. It encompasses a single processor or multiple processors, which may be located in the same geographical zone or may, at least partially, be located in different zones and may be able to communicate together.

Embodiments of the presently disclosed subject matter are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the presently disclosed subject matter as described herein.

<FIG> illustrates an embodiment of a system for detecting human drowning, for instance in a pool. The system may comprise at least a pool unit <NUM>, a remote alarm unit <NUM>, and a control unit <NUM>.

The remote alarm unit <NUM> may be located anywhere in a residence of a user or outside of it.

The system may comprise multiple remote alarm units. Such remote alarm units may be a part of the system (such as a remote alarm unit which is specific to the system), or may be units that are used for other applications or with other products and that the pool unit is paired with during the installation process or later.

The control unit <NUM> is operable on a processing unit and may comprise a control application. The control unit <NUM> may be installed e.g. on a smart phone, tablet, or any other computer, or on a dedicated control unit. This list is not limitative. The control unit <NUM> may also be part of the pool unit <NUM>. It may allow a user to control or to interact with the pool unit <NUM>.

An embodiment of the components that may be part of the pool unit's <NUM> is illustrated in <FIG>. The pool unit <NUM> may comprise a central processing unit <NUM>. The central processing unit may comprise for instance a CPU or multiple CPUs, a memory, CPUs (Graphical Processing Units), DSPs, FPGAs, other processing units, and a storage disk.

The pool unit <NUM> includes at least one video camera <NUM>. According to some embodiments, the system's cameras may enable a full visual coverage of the pool. According to some embodiments, each camera covers a different part of the pool.

For small pools, a single camera with for example at least a <NUM> degrees of horizontal angle and at least <NUM> degrees of vertical angle may be used (these numbers being not limitative), while placing the pool unit in one of the pool's corners.

For midsized pools, two cameras may be used in a back-to-back manner, while locating the pool unit in the middle of one of the long edges of the pool.

For larger pools, more cameras may be used, while some may be considered as extensions, and may be located in different locations of the pool, so that all cameras together may supply full coverage of the pool.

The system may also comprise a local alarm <NUM> (such as a speaker) for alerting on drowning risks.

The system is fed with power by a power supply <NUM>. According to some embodiments, it is supplied with power from an external source. According to some embodiments, the system may include solar panels and/or batteries.

The system may include a local control panel <NUM> that may include a small screen for user notifications, and may allow user input. In order to communicate with outer devices, like a remote alarm, or any remote control application, the system may comprise a communication unit <NUM>, which may be a wireless communication unit that may use any wireless communication technology, such as Wi-Fi, or Bluetooth.

While the system runs a video analytics method (embodiments of which will be described afterwards), the pool unit may also comprise a mechanical float <NUM> that may be used to indicate whether there was an entrance to the pool. Such information may be used as complementary information to the insights of the video analytics method, or as the only source of information, e.g. in situations of low visibility (dirty water or darkness) or occlusions.

As illustrated in <FIG>, the system may be used both in upper-ground and/or underground pools. <FIG> illustrates an embodiment of a pool unit for underground pools that uses solar panels. <FIG>illustrates an embodiment of a pool unit for underground pools that uses auxiliary electricity. <FIG>illustrates an embodiment for a pool unit for aboveground pools. These examples are not limitative.

<FIG> illustrates an embodiment of a method for drowning detection that may be performed by the system for drowning detection. The system senses underwater images (step <NUM>) from the video camera/s. Video frames from each camera may be used as input for the drowning detection method (step <NUM>), embodiments of which are described below. The drowning detection method may output warnings or alarms (step <NUM>). These warnings or alarms may be at different levels of severity. As a consequence, the system may output different types of alarms/messages in the local speakers, and if applicable, in the local control panel, and/or in the remote control units and/or in the remote alarm units.

Some embodiments of the drowning detection methods will now be described. As illustrated in <FIG>, in order to detect drowning events, the method may comprise analyzing a sequence of video frames, which may represent underwater images taken by the cameras of the system. Processing steps, such as detection, comparison, tracking of elements may for instance be performed by the central processing unit of the pool unit <NUM>.

The method comprises a step <NUM> of detecting which image parts comprise humans underwater, and which do not. This detection may be applied on part of the video frames and may be applied in different resolutions of the images at different times and at different image parts. The detection applied on a video frame may output a confidence level for the presence of humans at each image part. This detection may output humans-in-water candidates in the images, which comprise parts of the image that are detected as comprising a human. According to some embodiments, this detection uses at least a machine learning algorithm, which may be trained with a training set of underwater images.

Each humans-in-water candidate may be tracked in the video frames using a visual tracking algorithm (step <NUM>). The tracking method may be based for instance on Kalman filter, Partical Filters, mean shift or any other visual tracking method. At each time, the chance for each track to be associated with a drowning event is estimated (step <NUM>). Accordingly, the drowning risk is estimated based on non-movement detection, while the body is being fully submerged in water.

Embodiments of humans-in-water detection algorithms that detect humans from underwater images, in particular from images obtained by a single camera, will now be described. Detecting humans submerged, or partly submerged, in water (as opposed to pedestrian detections, for instance), may comprise recognizing humans at a variety of postures.

The humans-in-water detection/recognition methods may be based on machine learning algorithms that are pre-trained.

<FIG> is a general illustration of a possible training stage of a humans-in-water object detection/recognition algorithm <NUM>. This algorithm may be pre-operable on a processing unit that is not part of the pool unit (an embodiment is a personal computer or a cluster of computers), or can partly run on different computers and then further run on the pool units' processing unit. The algorithm <NUM> may be fed with a training set of underwater images. This training set may comprise "positive" examples <NUM>, which means that the underwater images comprise humans-in-water, and a set of "negative" examples <NUM> that may comprise any other images, such as underwater images without humans and underwater images with non-human objects.

According to some embodiments, the algorithm <NUM> outputs after the training stage a model or a plurality of models for humans-in-water. These models may be used to detect humans-in-water candidates in the underwater images. These models may comprise one or more visual representations (such as a particular distribution of pixels) that indicate the fact that a human is present in the underwater image.

The algorithm <NUM> may in particular comprise a machine learning algorithm. This machine learning algorithm may use any SVMs (Support Vector Machines), any neural network based technology, including the deep learning technology, CNN (Convolutional neural networks), Decision Trees/Forests based technologies, Bag-of-words based technology, HOG (Histogram Of Gradients) based technology, DPM (Deformable Parts Model) based technology, or any other classification/modelling technology that aims to distinguish between different types of objects. It may use one or more of the abovementioned technologies.

The output humans-in-water models may model explicitly, or implicitly, multiple sub-categories, each associated, for instance, with a different human posture (as detailed later) or any other type of subcategories. Other embodiments may be subcategorization by age or gender.

According to some embodiments, the machine learning algorithm is trained based on HOG (Histogram Of Gradients) and SVM. According to some embodiments, the machine learning algorithm is trained using DPM to be used as a detector.

According to some embodiments, a first machine learning algorithm is used to identify human-in-water candidates in the image. This first machine learning algorithm scans all the parts of the image with a sliding window detector in order to identify the humans-in-water candidates. Then, another machine learning algorithm, such as an object classification convolutional neural network, is applied only on the candidates/proposals that the first machine learning algorithm outputs (using a pre-defined threshold score). The object classification convolutional neural network may also be first trained using a training set of underwater images.

According to another embodiment, the object classification convolutional neural network performs both the identification of humans-in-water candidates, and the classification said candidates. Further explanations will be provided with reference to <FIG>.

<FIG> generally sketches a possible machine learning based humans-in-water detector <NUM>, which comprises at least a machine learning algorithm for detecting humans-in-water candidates in the underwater images. This machine learning algorithm may have been trained as explained e.g. in reference to <FIG>.

The detector <NUM> may receive as input at least an input underwater image <NUM>, which was taken by the underwater camera <NUM>. The detector <NUM> may use at least a humans-in-water model which is the output of the training phase detailed above (step <NUM>, which is the same as step <NUM> of <FIG>). According to some embodiments, the detector <NUM> may compare each part of the underwater image with the humans-in-water model in order to detect humans-in-water candidates.

The output of the detection method may be a set of humans-in-water candidates <NUM>, each associated with a location, and possibly other information, such as a grade associated with the detector's certainty, and possibly the association to a sub-category, associated, for instance, with posture (as detailed later).

There may be spatial overlaps between the candidates. The same candidate may be associated with more than one sub-category. The candidate's location may be described by a rectangle (associated with the bounding-box of the human candidate), by a silhouette, by a binary bitmap, or any other spatial description.

<FIG> illustrate the training and detection stages of an embodiment of a detector for which the humans-in-water model comprises different subcategories. References <NUM>, <NUM> and <NUM> are similar to references <NUM>, <NUM>, and <NUM>, respectively.

In this embodiment, the output of the training stage is a humans-in-water model which comprises a plurality of models, each model being associated to a different subcategory, with particular characteristics for each subcategory as illustrated by reference <NUM>. An example of subcategory is a posture of the human in water. Thus, the model of a given subcategory can represent how the human would appear in the underwater image for this given subcategory (e.g. different models can be created for different postures, or for different genders of the humans, etc.).

As discussed above in the context of <FIG>, the machine learning algorithm that is used for building the plurality of models may also be based on the SVM (Support Vector Machines) technology or any neural network based technology, including the deep learning technology, CNN (Convolutional neural networks), Decision Trees/Forests based technologies, Bag-of-words based technology, HOG (Histogram Of Gradients) based technology, DPM (Deformable Parts Model) based technology, or any other classification/modelling technology.

In <FIG>, the underwater image is compared to each of the models, wherein each model is associated to a subcategory. References <NUM> and <NUM> are similar to references <NUM> and <NUM>, respectively. Reference <NUM> is similar to reference <NUM>, but as illustrated, a plurality of human-in-water models is used. The output of the detector <NUM> when applied on an input underwater image may comprise a set of candidates for humans-in-water locations, while associating each candidate to a model of a particular subcategory.

According to some embodiments, the machine learning algorithm which attempts to detect humans-in-water candidates in the underwater is trained based on the input of a training set of underwater images clustered to subcategories. In particular, the underwater images of the training set which comprise a human ("positive" examples) may be pre-clustered by subcategory. As shown in <FIG>, the positive training images <NUM> may be divided using a clustering method <NUM> into sets, each set associated with a subcategory <NUM>. Embodiments for pre-clustering methods are described in the context of <FIG> below. The pre-clustered images may then be fed to the detection algorithm, in order to construct the different subcategories' models (<NUM>, which is the same as <NUM>).

According to some embodiments, when the training algorithm attempts to create a model representing each subcategory, it may use also the negative examples (<NUM>, which is the same as <NUM>). The clustering and modelling may be performed in an iterative manner (the dotted arrow <NUM> represents the option for the method to be iterative). For instance, using initial pre-clustering results, subcategory models are created. Then, all positive examples may be assigned to a subcategory by the model it best fits, then the models are re-trained. This may be repeated for a few finite number of iterations.

As illustrated in <FIG>, the subcategories may be implicitly or explicitly associated with posture of the human in water. An explicit association includes the fact that the candidates were explicitly associated to a posture, e.g. by a human annotator. An implicit association includes the fact that an algorithm that is based on certain descriptors extracted from the underwater image automatically clustered the data, wherein the resulted clustering turns out to be associated with posture.

The expression "posture" here refers to the way the human appears in the 2D projection in the image plane (and not to the 3D posture of the human in the 3D space). Examples of posture include e.g. appears horizontally in the image plane with head to the right and feat to the left, appears vertical in the image plane with head down and legs up, head facing camera and legs occluded, etc..

<FIG> illustrates an embodiment for a method for pre-clustering the images (an embodiment for how step <NUM> may be implemented). As mentioned above, according to some embodiments, the underwater images of the training set may be manually annotated as positive (including human) or negative ones (not including a full human). According to some embodiments, the positive examples may be further annotated by a manual mark of the location of the human in the image. This additional information can be used in the clustering procedure. The annotation may be, for instance, in the form of a rectangle that is a bounding-box of the human. In such a case the pre-clustering method may use the aspect ratio of the bounding box and/or the visual data in order to divide the examples into clusters. Another embodiment is that each positive example is manually pre-annotated by a subcategory, associated, for instance, to a posture (The user defines for each image to which subcategory the image belongs). In this case the clustering is a manual one with no algorithm involved. Another option for annotation is that manual annotations are provided by some details on the location of the body parts (without explicit posture annotation). According to some embodiments, the annotations may define a set of points that are each associated with a certain joint of the body. For example, a point for each elbow, knee, shoulder, etc (this list being not limitative". Then, a vector representing this point may be built to represent each image. Such annotations provide information that allows to then automatically pre-cluster the underwater images of the training set by the human posture.

Reference <NUM> shows an embodiment of annotation in which <NUM> points selected by a user describe together a "skeleton" of the human body. The positive examples <NUM> may be each annotated that way, and using these annotations, an automatic pre-clustering may be performed (step <NUM>), before training the object detection/recognition algorithm for building models for each subcategory using this pre-clustering.

Embodiments for pre-clustering the positive underwater images, using the "skeleton" annotations, comprise e.g.:.

According to some embodiments, pre-clustering can be used to pre-cluster the underwater images depending on the age and/or gender of the human(s) appearing in the underwater images. Similar pre-clustering methods to those described above can be used.

As already mentioned, the object detection/recognition algorithm may output a plurality of humans-in-water candidates per image, which may be overlapping. These humans-in-water candidates may each belong to a different subcategory (such as a posture). Each candidate may be associated to a confidence grade (indicating for instance the pertinence of the association of the model of a subcategory with respect to the image part covered by the human-in-water candidate). This confidence grade may be computed by the object detection/recognition algorithm. This is illustrated in step <NUM>, which is similar to step <NUM>. In step <NUM>, the various humans-in-water candidates that were detected for each subcategory for a given image are illustrated.

In order to cope with the overlap of the humans-in-water candidates, it is possible to unify similarly located candidates to serve as one candidate (step <NUM>). According to some embodiments, a known per se method called "nms" (non-maximal-suppression) may be used. The "nms" method selects only the candidates that have the maximal confidence grade in a certain neighbourhood, and throws away all other candidates. According to other embodiments, it is possible to recognize a set of highly overlapping candidates and to replace them by one candidate that is created by an average location. In particular, the location of the new candidate may be the center of gravity of the overlapping candidates, while weighting each candidate by its associated confidence grade.

The procedure of unifying similarly located candidates (overlapping candidates) may be performed for all humans-in-water candidates in an image as one set (that is to say even if they are associated to different subcategories), or for each subcategory separately. The latter option is illustrated in step <NUM>.

Different object detection and recognition algorithms have different detection/classification accuracy and require different computational resources. Usually accuracy comes on the cost of speed and vice versa. Therefore, when processing should be performed in real-time and the computation resources are limited, one has to consider both aspects of accuracy and processing time when selecting the algorithm to use. An embodiment comprises combining two or more detection/recognition algorithms and using them in a sequential manner: use a less resources demanding algorithm for scanning the whole data, and use a more accurate but more resource demanding algorithm in a later processing stage, after some filtering was done by the less demanding algorithm.

<FIG> illustrates such an embodiment. In this embodiment, a first object detection algorithm (machine learning algorithm) is used to scan all the parts of the image with a sliding window detector in order to identify the humans-in-water candidates (step <NUM>). As mentioned, each candidate may be given a confidence grade. Examples of such object detection algorithms were already provided.

Then, a second machine learning algorithm, such as an object classification algorithm, may be applied only on the candidates that the first machine learning algorithm outputs (step <NUM>). This second machine learning algorithm may be chosen as more accurate than the first machine learning algorithm. Revised and improved confidence grades are obtained for each candidate (step <NUM>). This may further allow filtering-out candidates that are recognized with high-confidence (that is to say above a given threshold) by the second machine learning algorithm as not being humans-in-water (step <NUM>, resulting output <NUM>).

A (not limitative) embodiment for such a combination is that the first algorithm used as the first stage is a HOG based detector using SVM, with or without the DPM scheme and the second algorithm is one based on the deep learning technology. Another (not limitative) embodiment is that both algorithms are based on the deep learning technology, while the first uses a certain number of layers in the neural network, and the second one uses a deeper neural network (with more layers, and maybe wider layers).

As described above, the subcategorization, e.g. by posture may serve the humans-in-water detection stage of the method by making it more accurate.

This subcategorization may also be used in the step of drowning detection. <FIG> illustrates in steps <NUM>, <NUM> the fact that humans-in-water candidates may be obtained for each image, wherein the humans-in-water candidates are divided by subcategory. These steps were already commented with respect to the previous embodiments.

According to an embodiment, a multi-object tracker may associate each humans-in-water candidate with an active track or a new track. The track may track objects that change in time and in location.

Since one of the system's aim is drowning detection, it is of interest to recognize situations in which a human is submerged and does not move. Therefore, the method may be interested in tracks in which the location and posture did not change over time. This may be accomplished by assigning different tracks for different postures <NUM>. For each active track it may be possible to estimate its "amount" of movement at a certain temporal window <NUM>, and in case of non-movement, or a movement smaller than a certain pre-defined for a pre-defined amount of time that is associated with a drowning risk, the method may output the severity of the event, and the system may output a warning, or an alarm, depending on the severity.

The different steps (as described in the various embodiments) of identifying the humans in water candidates in the images, tracking said candidates and detecting a risk of drowning may be performed based on the images taken by a single camera. If a plurality of camera are used (for example - but not limited to - when each camera covers a different zone), these different steps may be performed for each camera separately. The system may then aggregate the different drowning risks computed for each camera for taking the appropriate safety measure.

Claim 1:
A method of detecting human drowning, the method comprising the following steps performed by a processing unit (<NUM>):
∘ detecting (<NUM>) humans in a sequence of underwater images taken by a single camera (<NUM>), for identifying humans-in-water candidates in the images, said detecting comprising detecting which parts of the images comprise humans underwater, and which do not, wherein the part of the images detected as comprising humans underwater correspond to said humans-in-water candidates in the images, wherein said detecting comprises using a first deep learning neural network scanning the sequence of underwater images using a sliding window and providing proposals for the humans-in-water candidates in the sequence of underwater images, and a second deep learning neural network applied only on the proposals, wherein a number of layers of the second deep learning neural network is larger than a number of layers of the first deep learning neural network,
∘ tracking (<NUM>) the humans-in-water candidates throughout this sequence, and
∘ detecting (<NUM>) a lack of movement of the humans-in-water candidates, in order to detect human drowning risk.