Patent Description:
Access control systems are used in a variety of applications including structures, buildings and or components including safes, subway turnstiles, child proof storage containers, and many other applications. In the, non-limiting, example of buildings, many such structures must be secured in the sense that the identification and number of people entering and exiting a building at any given moment in time should be known. One known way in achieving this task is to assign a badge to all individuals requiring access. Each human is then required to perform a hard badge-in task at a reader located proximate to any entry point. In one example, the badge may be identified by the reader via a magnetic strip. Another example is reading a badge using RFID. Unfortunately, such a process requires each human to, for example, swipe their badge separately before entry is allowed. This task can be time consuming.

More current access control systems utilize smartphones in place of badges. A key technology behind such use of smartphones is Near Field Communications (NFC) which allows short range communication. With this application, both the smartphone and the local access control reader must have NFC hardware. Other options may include a Human Interface Device (HID) of a reader capable of detecting, for example, a twisting of a smartphone in front of the reader in a controlled fashion to show intent. However, both the smartphone and the reader must be capable of independently detecting the intent. Moreover, current methods still require the user to retrieve the smartphone and perform specific acts with the smartphone. Such retrieval and/or action can be frustrating for the user and time consuming.

<CIT> discloses systems and methods for providing sensor based authentication of a user's identification that may be used to control access.

<CIT> discloses a handheld computer system and methodology for human hand gesture recognition and for user authentication by means of motion gestures.

<CIT> discloses a portable electronic device for controlling a remotely positioned electronic arrangement, having processing circuitry for transmitting a functional command together with an identity of an authenticated user to the electronic arrangement.

Improvements in access systems that may further optimize ease of operation with, or without, reduced components is desirable.

The invention relates to a method of training a software-based application for a gesture-based access control system, as defined by the appended independent claim <NUM>. The dependent claims define further preferred embodiments of the invention.

A method of training a gesture-based access control system according to one, non-limiting, embodiment of the present disclosure includes repetitiously performing an intentional gesture indicative of the intentional gesture type by a user of a mobile device, wherein the intentional gesture is representative of an intent of the user to gain access; and forming a model of the selected intentional gesture type from the repetitious performances of the intentional gesture, and by a software-based application of the gesture-based access control system, wherein the model is applied by the software-based application to confirm future performances of the intentional gesture.

Additionally to the foregoing embodiment, the method includes selecting the intentional gesture type from a preprogrammed library of a plurality of intentional gesture types prior to the repetitious performance.

In the alternative or additionally thereto, in the foregoing embodiment the method includes applying at least one machine learning algorithm to analyze data collected from the repetitious performances of the intentional gesture.

In the alternative or additionally thereto, in the foregoing embodiment, the method includes applying at least one machine learning algorithm to analyze data collected from the repetitious performances of the intentional gesture.

In the alternative or additionally thereto, in the foregoing embodiment, the data is collected from at least one detection system of the mobile device.

In the alternative or additionally thereto, in the foregoing embodiment, the data includes motion data detected by an inertial measurement unit (IMU) sensing system of the at least one detection system.

In the alternative or additionally thereto, in the foregoing embodiment, the data includes audible data detected by a microphone of the mobile device.

In the alternative or additionally thereto, in the foregoing embodiment, the intentional gesture is tapping by the user upon the mobile device.

In the alternative or additionally thereto, in the foregoing embodiment, the software-based application prompts the user to select the intentional gesture type via a human interface device (HID) of the mobile device.

In the alternative or additionally thereto, in the foregoing embodiment, the software-based application notifies the user upon completion of the model based on confidence, and via a human interface device (HID) of the mobile device.

In the alternative or additionally thereto, in the foregoing embodiment, the preprogrammed library includes at least one of tapping on the mobile device by the user, knocking on a door by the user, and a user voice command.

In the alternative or additionally thereto, in the foregoing embodiment, the software-based application is configured to apply a performance of the intentional gesture to confirm user intent and authenticate the user.

In the alternative or additionally thereto, in the foregoing embodiment, the method includes analyzing a frequency of the intentional gesture by the software-based application to authenticate the user, wherein the intentional gesture is one of a tapping and a knocking.

However, it should be understood that the following description and drawings are intended to be exemplary in nature and non-limiting.

Referring to <FIG>, a gesture-based access control system <NUM> is illustrated in one, non-limiting application, of a door <NUM> providing user access into, and out of, a building, structure, room, or the like. In this embodiment, the access control system <NUM> is adapted to unlock the door upon a detected, intentional, gesture made by a user <NUM> (e.g., human) desiring access. Although the present application is applied to the door <NUM>, it is contemplated and understood that the access control system <NUM> may also apply to anything requiring access control including, for example, computers, subway turnstiles, safes, child proof storage compartments, and others. As will become more apparent, the intentional gesture may be a device-free gesture (see arrow <NUM> in <FIG>) in some embodiments, or a device gesture (see arrow <NUM> in <FIG>) in other embodiments.

Referring to <FIG>, and in one embodiment, the access control system <NUM> includes a lock, or access, assembly <NUM>, a mobile device <NUM> carried by the user <NUM>, and a wireless interface <NUM>. The mobile device <NUM> is adapted to wirelessly communicate with the lock assembly <NUM> over the wireless interface <NUM>. The lock assembly <NUM> may include a latch <NUM> (e.g., deadbolt), a driver <NUM>, a controller <NUM>, and a receiver <NUM> that may be a transceiver with bi-directional communication capability, and that includes an antenna. The receiver <NUM> is configured to receive a wireless access, or command, signal (see arrow <NUM>) over the wireless interface <NUM> and from the mobile device <NUM>. The access signal <NUM> is sent to the controller <NUM>. The controller <NUM> may process the signal <NUM>, and based on the signal, initiate the driver <NUM> to move the latch <NUM> from a no-access state to an access state (i.e., locked and unlocked positions). In one embodiment, the access assembly <NUM> is an access reader (e.g., RFID reader). Examples of the signal <NUM> may be Bluetooth, Wifi, or other communication signals that may be short range. The access assembly <NUM> may be a local access assembly <NUM>, and is generally located proximate to the door, or other component, whose access the assembly <NUM> is adapted to control.

The controller <NUM> may be any combination of one or more of a central processing unit (CPU), multiprocessor, microcontroller unit (MCU), digital signal process (DSP), application specific integrated circuit, and others capable of executing software instructions, or otherwise controllable to behave according to predetermined logic. In one example, the driver <NUM> is an electric motor with a relay operated by the controller. In another example, the driver <NUM> is an electromagnetic driver. The wireless interface <NUM> is any current or future wireless interface allowing communication between the mobile device <NUM> and the lock assembly <NUM>. Non-limiting examples of the wireless interface <NUM> include Bluetooth, Bluetooth Low Energy (BLE), Radio Frequency Identification (RFID), Near Field Communication (NFC), any of the IEEE <NUM> standards, and others.

In one embodiment, the mobile device <NUM> includes a transmitter <NUM> that may be a transceiver having an antenna, a controller <NUM>, and at least one detection system (i.e., three illustrated as <NUM>, <NUM>, <NUM>). The at least one detection system may include an inertial measurement unit (IMU) sensor system <NUM>, an environment detection system <NUM>, an internal activity (i.e., usage) notification module <NUM>, and others for generally determining motion, position, location, and usage of the mobile device <NUM> relative to the user <NUM>. Non-limiting examples of the mobile device <NUM> include a smartphone, a mobile phone, a key fob, a wristwatch (i.e., smart watch), and other similar devices typically carried by the user <NUM>.

The controller <NUM> of the mobile device <NUM> includes a processor <NUM> and a storage medium <NUM>. Optionally, the processor <NUM> is any combination of one or more of a central processing unit (CPU), multiprocessor, microcontroller unit (MCU), digital signal processor (DSP), application specific integrated circuit, and others capable of executing software instructions or otherwise controllable to behave according to predetermined logic. The storage medium <NUM> is, optionally, any combination of read and write memory (RAM) and read only memory (ROM). The storage medium <NUM> may also include persistent storage, which can be any single one or combination of solid state memory, magnetic memory, or optical memory storing a computer program (i.e., application) with software instructions.

In one embodiment, and similar to the controller <NUM> of the mobile device <NUM>, the controller <NUM> of the lock assembly <NUM> may include a processor <NUM> and a storage medium <NUM>. Optionally, the processor <NUM> is any combination of one or more of a central processing unit (CPU), multiprocessor, microcontroller unit (MCU), digital signal processor (DSP), application specific integrated circuit, and others capable of executing software instructions or otherwise controllable to behave according to predetermined logic. The storage medium <NUM> is, optionally, any combination of read and write memory (RAM) and read only memory (ROM). The storage medium <NUM> may also include persistent storage, which can be any single one or combination of solid state memory, magnetic memory, or optical memory storing a computer program (i.e., application) with software instructions. It is contemplated and understood that in one embodiment, the controller <NUM> may not include a storage medium <NUM>, and may only include control circuitry capable of receiving the signal <NUM> from the mobile device <NUM> as a command signal that initiates actuation of the lock assembly <NUM>.

The gesture-based access control system <NUM> may further include an application <NUM>. In one embodiment, the application <NUM> is software-based and is stored, at least in-part, in the storage medium <NUM> for retrieval and execution by the processor <NUM> of the controller <NUM>. The application <NUM> may include computer instructions <NUM>, and a database of preprogrammed data. For example, the preprogrammed data includes credential data <NUM>, and scenario data <NUM>. In one embodiment, the scenario data <NUM> is indicative of a 'compound' motion by the user <NUM> that may not necessarily include the gesture, but is dependent upon (i.e., a function of) the carrying location of the mobile device <NUM> on the user <NUM>.

In another embodiment, the application <NUM> may at least in-part be stored in at least one storage medium contained in a cloud (i.e., remote server) and executed at least in-part by at least one processor of the cloud.

For reasons of clarity, the term "intentional gesture" as used herein is an act (e.g., physical motion) performed by the user <NUM> to gain access. In one example, the access gained may be through a door <NUM> (see <FIG>), but may also be access into any physical structure and/or electronic systems (e.g., computer). For purposed of this disclosure, examples of an intentional gesture may include a device-free gesture, a device gesture, and an inherent gesture.

The term "device-free gesture," refers to an intentional gesture that generally does not physically include the mobile device <NUM> (see gesture <NUM> in <FIG>). For example, if the device-free <NUM> made by the user <NUM> is the waving of a right hand <NUM>, the mobile device <NUM> is not in the right hand <NUM> but may be located anywhere else on the person of the user <NUM>. In contrast, the term "device gesture," (see gesture <NUM> in <FIG>) means the mobile device <NUM>, itself, is being used as part of the intentional gesture. In the present example, the device gesture <NUM> would include the waving of the mobile device <NUM>. More specifically and in line with the present example, the mobile device <NUM> would be in the right hand <NUM> being waved (see <FIG> and <FIG>). Lastly, the term "inherent gesture" (see gesture <NUM> in <FIG>) is the gesture applied as part of a seamless access control system. That is, the typical act of, for example, opening a door (or typical motion(s) made toward the preparation of opening the door) is the gesture. The inherent gesture is "intentional" in the sense that the user <NUM> intends to gain access. Specific examples of the inherent gesture may be reaching for a door handle, or pulling upon a door handle.

Determination of motion (i.e., the compound motion) of the mobile device <NUM> is needed to recognize an intentional gesture made by the user <NUM> through differentiation of one or more motions made by the user simultaneously. The determination of the position and/or location of the mobile device <NUM> relative to the user <NUM> may assist in the differentiation of multiple motions made by the user <NUM> from the measured compound motion of the mobile device <NUM>. Alternatively, or in addition to, determining the location of a mobile device <NUM> with respect to the user <NUM> may be advantageous when two access assemblies <NUM> if respective doors <NUM> are positioned closely together. In this scenario, knowing the location of the mobile device <NUM> would prevent, or reduce the chances, of the user <NUM>, via the device-free intentional gesture, gaining access through the wrong door.

The inertial measurement unit (IMU) sensor system <NUM> may include one or more of an accelerometer <NUM>, a gyroscope <NUM>, and others adapted to detect acceleration, and thus movement, in at least one dimension, and optionally three dimensions. The environment detection system <NUM> may include one or more of a visual camera <NUM> (i.e., computer-vision system), a temperature sensor, <NUM>, a light sensor <NUM>, and a proximity sensor <NUM> adapted to at least improve a level of confidence when differentiating the compound motion to determine if a device-free intentional gesture is being made by the user <NUM>.

The internal activity notification module <NUM> may also contribute toward the optimization of confidence levels, and may be part of the application <NUM> or may be a separate computer software instruction. For example, the activity notification module <NUM> may notify the application <NUM> that the user <NUM> is texting via the mobile device <NUM>, or is conducting a phone conversation. When differentiating the compound motion, the application <NUM> may then attribute part of the motion toward, for example, the texting activity. In one embodiment, and depending upon how the information data is processed by the application <NUM>, the visual camera <NUM> may be part of the IMU sensor system <NUM> (i.e., taking multiple pictures to determine motion), and/or may be part of the internal activity notification module <NUM> (i.e., the user <NUM> is undergoing the activity of taking photographs for pleasure).

In one embodiment, the visual camera <NUM> is adapted to detect movement via the capturing of images of surroundings and analyzing differences in the images over time. The temperature sensor <NUM> is adapted to measure temperature. In one embodiment, temperature data is indicative of, at least in-part, the body temperature of the user <NUM>. For example, if the mobile device <NUM> is in a rear pocket <NUM> (see <FIG>) of clothing worn by the user <NUM>, the temperature data may be associated with a temperature that is higher than if the mobile device <NUM> were located in a purse or backpack worn by the user <NUM>. The proximity sensor <NUM> is adapted to determine how close the mobile device <NUM> is to the user <NUM>. For example, the mobile device <NUM> may be resting on a desk, may be in a back pocket <NUM>, may be in a purse, or may be in a backpack. The proximity sensor <NUM> may also be used to determine if a substantial portion of the user <NUM> is located between the sensor <NUM> and the access assembly <NUM>, which may cause a degree of attenuation of signals between the assembly <NUM> and the mobile device <NUM>.

The light sensor <NUM> is adapted to measure the level of light adjacent to the mobile device <NUM>. Light data sent to the processor <NUM> from the light sensor <NUM> may be indicative of the location of the mobile device <NUM> at the time of gesturing by the user <NUM>. For example, the mobile device <NUM> may be in the rear pocket <NUM> of clothing worn by the user <NUM>.

In operation, the IMU sensor system <NUM> enables the identification of gesture based intent, and the environment detection system <NUM>, and optionally the activity notification module <NUM> function to boost the reliability of the intentional gesture identification. In one example, this is achieved by the fusion of information gained from the systems <NUM>, <NUM>, and module <NUM> by the application <NUM> and use of machine learning algorithm(s) and/or the preprogrammed scenario data <NUM>. Referring to <FIG>, a method of determining a location and/or position of a mobile device <NUM> with respect to the user <NUM> includes, at block <NUM>, the motion device <NUM> activity being in standby, or otherwise blocked.

At block <NUM>, the IMU sensor system <NUM> detects a periodic movement (i.e., the compound motion) and sends the information to the controller <NUM>. At block <NUM>, the application <NUM> determines that at least a portion of the compound motion is characteristic of walking via at least one algorithm, and at least a portion of the preprogrammed scenario data <NUM>. At block <NUM>, the temperature sensor <NUM> and/or the light sensor <NUM> of the environment detection system <NUM> sends information (i.e., confirmation parameter data) to the controller <NUM> that is used by the application <NUM>, to determine that the mobile device <NUM> is in, for example, a back pocket or a backpack (i.e., the light sensor <NUM> detects a dark environment). Moreover, the IMU sensor system <NUM> may also assist in detecting the relative position of the mobile device <NUM>. For example, the angle of the mobile device <NUM> with respect to the ground, or floor surface, may be indicative front pocket verse back pocket location, etc.. At block <NUM>, the activity notification module <NUM> may provide information to the application <NUM> indicative of the current use (e.g., texting) of the mobile device <NUM> by the user <NUM>. Such current use may provide indications of the likely position of the mobile device <NUM> (i.e., vertical, horizontal, or positions there-between) and/or mobile device motions that are part of the compound motion which may ultimately be differentiated from the intentional gesture. To accomplish blocks <NUM> and <NUM>, the application <NUM> may apply an algorithm and/or the preprogrammed scenario data <NUM>.

Referring to <FIG> and in operation, according to the present invention the application <NUM> may include training instructions (i.e., setup or calibration instructions) communicated to the user <NUM> via a human interface device (HID) <NUM> (see <FIG>) of the mobile device <NUM>. The training instructions may instruct the user <NUM> to perform a variety of motions with the mobile device <NUM> carried by the user <NUM> in various locations (e.g., back pocket, front pocket, left hand while right hand is gesturing, and others), or ways (e.g., backpack, purse, and others), and/or while performing certain activities with the mobile device <NUM> (e.g., texting, conversing, and others). While the user <NUM> performs the various motions and/or routines, the application <NUM> may build, and thus preprogram, the scenario data <NUM> utilizing information received from the at least one of the IMU sensor system <NUM>, the environment detection system <NUM>, and the internal activity notification module <NUM>.

For example, the application <NUM> may instruct the user <NUM> to walk with the mobile device <NUM> in the rear pocket <NUM>. The motion and other parameters are then detected by at least one of the systems <NUM>, <NUM>, and the module <NUM>, and the resulting information is preprogrammed as part of the scenario data <NUM>. As part of another event, the application <NUM> may then instruct the user <NUM> to perform the same walk with the mobile device <NUM> in the same location, but while performing a chosen gesture intended to cause the access assembly <NUM> to respond (i.e., unlock). Again, the resulting motion detected by one or more of the systems <NUM>, <NUM> and module <NUM> is recorded as part of the scenario data <NUM>. Similar instructions may progress with the user <NUM> relocating the mobile device <NUM> on his or her person and performing various movements with and without the gesturing. Upon completion of the training instructions, the scenario data <NUM> may generally resemble a matrix or array of data.

In one embodiment, the application <NUM> may include machine learning techniques and/or algorithms (e.g., deep learning). With machine learning algorithms, gesture recognition can be trained more and more to a given user's particular interactions. Moreover, by conducting a form of 'continuous' training, the application <NUM> has the ability to conform to a user's changing habits (i.e., possibly caused by an injury) over a period of time.

In one example, the application <NUM> may include machine learning algorithm(s) configured to determine, or confirm, user intent from explicit intent signal(s) generated by one or more of the detection systems <NUM><NUM>, <NUM>, and determine user authentication (i.e., the mobile device <NUM> actually belongs to the user <NUM>) by matching the intent signals against a user specific, pre-defined, pattern. The user intent and user authentication may be inferred from IMU signals, audio signals, RSSI (e.g., Bluetooth), and other data from, for example, from wearable mobile devices <NUM>. In another embodiment, while user intent may be confirmed by a number or pattern of knocks, user authorization may be confirmed by the intensity of the knocks, a delay between knocks, and/or a change of intensity from one knock to the next.

Referring to <FIG> and in one embodiment, the application <NUM> may include a training mode of operation. At block <NUM>, and via the HID <NUM>, the user <NUM> may select the training mode. In this mode, and at block <NUM>, the user <NUM> is prompted by the application <NUM> via the HID <NUM>, and may select, an intentional gesture type from a library of supported gesture types as part of the scenario data <NUM>. At block <NUM>, the user <NUM> is prompted by the application <NUM>, and the user <NUM> may perform, repetitions of the selected gesture type for intent. At block <NUM>, machine learning algorithm(s) are collecting and analyzing data from the repetitious performance of the selected gesture type to build a user specific model associated with selected gesture type and as part of the scenario data <NUM>. At block <NUM>, the machine learning algorithm(s) determine that that the user specific model is of sufficiently high quality and confidence, and the application <NUM> via the HID <NUM>, notifies the user <NUM> of model completion. Non-limiting examples of gesture types may include tapping by the user <NUM> on the mobile device <NUM> for a fixed number of times (i.e., a prescribed pattern, see <FIG>), a knock on the door <NUM>, a user specific voice command made into a microphone <NUM> of the mobile device <NUM> (see <FIG>), and other gesture types.

After the training mode of operation, the application <NUM> may enter into a deployment mode. In this mode, statistical machine learning techniques are deployed, via algorithms, which may be in, and supported by, a cloud <NUM> (i.e., a remote server, see <FIG>). In this example, at least a portion of the application <NUM> may be in the cloud <NUM>, and the cloud functions to build the user specific model. In one embodiment, the user specific model may be improved over time via the use of machine learning algorithms. In this way, specific users <NUM> become easier to identify over time. At block <NUM>, the user <NUM> may then perform a list of pre-trained gestures (i.e., preprogrammed into the application <NUM>) to signal intent and authenticate them.

More specifically, in the training mode of operation, data is collected reflective of specific actions enforced upon the user <NUM> for purposes of training. This may be considered as defining the ground truth of the `right way' of performing a gesture. Optionally, the application <NUM> may also collect data on how the specific actions is not to be performed to further enhance the learning.

Once the training mode is complete and the data is collected, algorithms are then trained with the data to extract the relevant information/features that detect if the specific action, or gesture, was performed and in the right way. The result is a trained model (i.e., the user specific model) that is then deployed.

Referring to <FIG>, a graph <NUM> having three portions 118A, 118B, 118C is illustrated that generally reflects one example of a modeling process wherein the gesture type may be tapping on the mobile device <NUM>. The X-axis of each graph portion 118A, 118B, 118C is over a common time duration. Graph portion 118A illustrates raw accelerometer data caused by movement of the mobile device <NUM> incurred during tapping. Graph portion 118B illustrates corresponding audio data. Graph portion 118B illustrates extracted features with the tapping confirmation highlighted with star symbols. The spike patterns and the time intervals between spikes are unique to the user <NUM> and may be used as the authentication (i.e., code).

Completion of the training and deployment modes produces the user specific detection model that serves both as gesture confirmation and a user authentication based on the observed signals from one or more of the detection systems <NUM>, <NUM>, <NUM>. The model also provides a confidence level in user authentication that may improve with further usage. This confidence level may be used to allow or deny access to, for example, building areas.

In one embodiment, the application <NUM> may rely on the observation that the device-free gesture (e.g., hand waving) produces minute periodic motion of the human body (i.e., a part of the compound motion) that can be captured using the IMU sensor system <NUM>, the environment detection system <NUM>, and/or the internal activity notification module <NUM> of the mobile device <NUM>. Machine learning algorithms are trained to distinguish the associated minute motion, indicative of the gesture, from other and more prominent body movements that may be observed during walking or conversing.

Optionally, the controller <NUM> of the mobile device <NUM> may receive data from the light system <NUM>. In one example, the light data may be applied to determine if the mobile device <NUM> is carried in a hand, or alternatively, in a pocket, backpack, or purse. The temperature sensor <NUM> of the environment detection system <NUM> may output temperature data to the controller <NUM> to determine if, for example, the mobile device <NUM> is in a hand or pocket, as oppose to in a backpack or purse. The temperature and/or light data may be applied as additional data toward the compound motion to increase matching confidence levels when the application <NUM> compares, or attempts to match, the compound motion to the preprogrammed scenario data <NUM>.

In one embodiment, the chosen device-free intentional gesture may be the waving of a hand <NUM> (see <FIG>) that is free of the mobile device <NUM>. That is, the mobile device <NUM> is located elsewhere on, or near, the user <NUM>. In other words, the user <NUM> is not required to retrieve his/her mobile device <NUM> to perform any device function or input. The user <NUM> need only perform the correct intentional gesture to gain access through, for example, the door <NUM>. Examples of other intentional gestures may include left-to-right motions of a human arm, up-to-down motions of the human hand <NUM>, a motion of the head and/or shoulders, or any other distinctive motion.

In one embodiment, the intentional gesture may be a secret gesture, thus further authentication between the mobile device <NUM> and the access assembly <NUM> is not needed. In this example, the access assembly <NUM> may be relatively simple, and need not be preprogrammed.

In another embodiment, the access assembly <NUM> may be preprogrammed to only accept command signals <NUM> that are entrained, or accompanied, with an authentication code generally preprogrammed into both controllers <NUM>, <NUM>. Thus the controller <NUM> is capable of matching a received authentication code from the mobile device <NUM> (i.e., part of signal <NUM>) to a code <NUM> preprogrammed into the storage medium <NUM>.

Referring to <FIG> and <FIG>, and during normal operation of the gesture access control system <NUM>; at block <NUM>, the controller <NUM> of the access assembly <NUM> may broadcast a beacon signal (see arrow <NUM> in <FIG>) via the transceiver <NUM>. In one example, the beacon signal <NUM> may be encoded as part of the authentication process between the mobile device <NUM> and the access assembly <NUM>. In one example, the broadcast beacon signals <NUM> may be of a Bluetooth radio type. In other examples, the signal <NUM> may be Wifi/cell radio or may be an audible frequency spectrum. It is further contemplated and understood that other ways of authenticating the mobile device <NUM> with the access assembly <NUM>, which are known by thus skilled in the art, may be applied while the novelty of the gesturing process is maintained.

At block <NUM>, the transceiver <NUM> of the mobile device <NUM> may receive the beacon signal <NUM> when generally within a prescribed range. Once received, at block <NUM>, the mobile device <NUM> generally initiates the application <NUM>. In another embodiment, the application <NUM> may not need to be initiated by a beacon signal. Therefore, in some applications, the access assembly <NUM> may not be adapted to broadcast a beacon signal.

At block <NUM>, when within a general vicinity of the access assembly <NUM>, and/or with the application <NUM> active, the application <NUM> may be accepting and processing compound motion data from the IMU sensor system <NUM> of the mobile device <NUM> to determine the activity of the user <NUM> (i.e., walking, conversing, standing still, and others), and other influencing data or information from the environment detection system <NUM>, and/or the internal activity notification module <NUM> to determine influential parameters such as the mobile device location, position and/or usage. At block <NUM>, the application <NUM> matches the compound motion data and influencing parameter data to the preprogrammed scenario data <NUM>, with a predetermined level of confidence, to determine if the user <NUM> is performing an intentional gesture (e.g., device-free intentional gesture) indicative of an intent to access.

At block <NUM>, and in one example, the user <NUM> may be walking with the mobile device <NUM> in a rear pocket, and while performing a device-free intentional gesture with the right hand <NUM>. At block <NUM>, the application <NUM> determines where the mobile device <NUM> is located on the user <NUM>, determines that the user <NUM> is walking, and determines that the device-free intentional gesture is being performed by comparing the compound motion and other influencing parameter data (e.g., light, temperature, and others) to the scenario data <NUM>. At block <NUM>, and after recognition of the device-free intentional gesture by the controller <NUM> of the mobile device <NUM>, the mobile device <NUM> broadcasts a command signal <NUM> to the access assembly <NUM>. At block <NUM>, the access assembly <NUM> actuates from a no-access state and to an access state, whereupon the door <NUM> may be opened by the user <NUM>.

In one embodiment, it may be a pre-condition that the user <NUM> is not walking before a gesture may be recognized or accepted by the mobile device <NUM>. In this embodiment, the accelerometer system and/or the gyroscope system of the mobile device <NUM> may be applied to confirm the user <NUM> is generally motionless except for the motion of the gesture itself.

Referring again to <FIG>, the beacon signal <NUM> broadcasted by the access assembly <NUM> via the transceiver <NUM> may be received by the controller <NUM>, via the transceiver <NUM>, and generally as a received signal strength indicator (RSSI). More specifically and as an optional embodiment, the gesture-based access control system <NUM> may further include an RSSI module <NUM> that may be software-based and part of the application <NUM>. In other embodiments, the RSSI module <NUM> may by a separate sensor system of the mobile device <NUM> that may include software and hardware.

In operation, the gesture-based access control system <NUM> may perform as described in blocks <NUM>-<NUM> (see <FIG>), except with the additional feature provided by the RSSI module <NUM>. More specifically, the beacon signal <NUM> received by the mobile device <NUM> at block <NUM> is also processed by the RSSI module <NUM> that is configured to detect periodic variations in signal strength indicative of the intentional gesture crossing through the signal <NUM> (i.e., near to and repetitiously crossing in front of the access assembly <NUM>). In one example, it may be an arm of the user <NUM> crossing back-and-forth in front of the access assembly <NUM>. In another embodiment, the placement of a hand of the user <NUM> on the access assembly <NUM> may also effect RSSI.

As described in block <NUM> above, the scenario data <NUM> may further include preprogrammed RSSI data indicative of the detected periodic variation in signal strength expected when the device-free gesture is performed. The RSSI module <NUM> may compare the measured periodic variation in signal strength to the preprogrammed RSSI data to further confirm, or increase a level of confidence, that the device-free gesture occurred.

In another embodiment, the scenario data <NUM> may only include the preprogrammed RSSI data. In this embodiment, the determination by the application <NUM> that the device-free gesture was performed may be based solely on the preprogrammed RSSI data. Therefore, the IMU sensor system <NUM> may not be required.

As previously described, the mobile device <NUM> may be located remotely from the immediate vicinity of the intentional gesture (i.e., device-free gesture <NUM>) being performed. For example, the mobile device <NUM> may be carried generally against the body of a user <NUM> (e.g., rear pocket) but not in the hand <NUM> performing the device-free gesture (see <FIG>).

Referring to <FIG>, a generally device-free gesture <NUM> may be performed by the user <NUM>, but with the mobile device <NUM> located in a user-carried containment <NUM>. Non-limiting examples of the containment <NUM> include a handbag (see <FIG>), a backpack (see <FIG>), and other containments adapted to store and/or carry personal items for the user <NUM> including the mobile device <NUM>.

In one embodiment, the containment <NUM> is adapted to be carried by a specific body component of the user <NUM>. For example, the handbag is carried by the hand <NUM> of the user <NUM> and the backpack is carried by the back, or torso, <NUM> of the user <NUM>. For high confidence detections of the device-free gesture <NUM>, the containment <NUM> is carried by the body component performing the device-free gesture <NUM> (i.e., intentional body gesture). For example, if the containment <NUM> is a handbag or purse, the hand <NUM> that grasps the handbag may perform the device-free gesture <NUM> thus carrying the handbag along with the gesturing hand.

The motion of the mobile device <NUM> is generally measured as previously described using at least the IMU sensor system <NUM>. In one scenario, the measured motion of the mobile device <NUM> may be a compound motion dynamically created by the user <NUM> walking as the user performs the intentional body gesture <NUM> (i.e., device-free gesture). In this scenario, the act of walking may cause the user <NUM> to swing the arm and hand <NUM> (i.e., a routine body motion, see arrow <NUM> in <FIG>) in forward and rearward directions. The swinging of the hand <NUM> carries the handbag <NUM> with it causing the mobile device to experience an associated routine containment motion (see arrow <NUM> in <FIG>).

Referring to <FIG> and in a continuation of the containment <NUM> example of a handbag, the intentional body gesture <NUM> may be the twisting of a wrist associated with the hand <NUM> of the user <NUM> that is grasping the handbag <NUM>. The intentional body gesture <NUM> creates an associated containment gesture (see arrow <NUM>). In one embodiment, the containment gesture <NUM> may be an amplification of the intentional body gesture <NUM>. In other embodiments, gesture <NUM> may be about the same as gesture <NUM> or may be different but expected.

The measured motion of the mobile device <NUM> is thus a compound motion that includes the containment gesture <NUM>, which is directly affiliated with the intentional body gesture <NUM>, and the routine containment motion <NUM> that is affiliated with the routine body motion <NUM>. Therefore, the compound motion is indicative of the routine body motion <NUM> and the intentional body gesture <NUM> multiplied by a parameter factor. The parameter factor may represent the type of containment <NUM> (i.e., backpack or handbag) and the position and location of the mobile device <NUM> with respect to the user <NUM> and the containment <NUM>. The parameter factor may be part of the scenario data <NUM>, and the environment detection system <NUM> may assist in determining the position and location of the mobile device <NUM> and the type of containment <NUM>.

In one embodiment, the intentional body gesture <NUM> is such that the associated containment gesture <NUM> is contrary to the routine containment motion <NUM>. For example, the direction of gesture <NUM> is traverse, or orthogonal to the direction of motion <NUM>. This will assist in higher levels of confidence through improved motion differentiation by the application <NUM>.

Referring to <FIG>, another example of a containment gesture <NUM> is illustrated wherein a handbag is shaken vertically. In this example, the intentional body gesture may be the repetitious lifting and lowering of the hand <NUM>.

Referring to <FIG>, another example of a containment <NUM> is illustrated as a backpack worn on the back, or torso, <NUM> of the user <NUM>. In <FIG> the containment gesture <NUM> may be caused by a twisting (i.e., the intentional body gesture <NUM>) of the torso <NUM>. In <FIG>, the containment gesture <NUM> may be caused by a bending at the waist of the user <NUM>. In <FIG>, the containment gesture <NUM> may be caused by a flexing left-to-right of the torso <NUM> or waist of the user <NUM>.

As previously described, determining the occurrence of a device-free gesture can be accomplished through the analysis of a measured compound motion of the mobile device <NUM> and other influencing parameters. For example, if the mobile device <NUM> is in a back pocket <NUM>, and a right hand <NUM> is performing the device-free gesture, the compound motion undergone by the mobile device <NUM> is analyzed as an indirect indication of the device-free gesture occurrence.

Referring to <FIG> and <FIG>, and in another embodiment, the mobile device <NUM> may be used to perform the gesture (i.e., a device gesture). In this example, the device gesture is generally measured directly as the motion of the mobile device <NUM>. However, it is still appreciated that the motion measured by the mobile device <NUM> may still be a type of compound motion.

For example, the device gesture (see arrow <NUM> in <FIG>) may generally be a generally horizontal waving of the mobile device <NUM>. If the user <NUM> remains perfectly still, other than performing the device gesture <NUM>, the mobile device <NUM> can measure the device gesture <NUM> directly and no motion differentiation of a compound motion is needed. However, if the user <NUM> is walking while performing the device gesture <NUM>, the walking motion will also be measured with the device gesture <NUM> thus producing a measured compound motion. That is, the walking motion creates a kind of noise that may interfere with a reliable interpretation of access intent.

The compound motion in this example may be analyzed as previously described with proper scenario data <NUM> established with the prescribed condition that the intentional gesture is a device gesture <NUM>. Other, non-limiting, examples of device gestures <NUM> may include waving the mobile device <NUM> in a substantially vertical direction in front of the access assembly <NUM> (i.e., an imitated swiping of an imaginary access card, see <FIG>), repeatedly moving the mobile device <NUM> toward and away from the access assembly <NUM> (see <FIG>), generally twisting the mobile device <NUM> by about ninety degrees in front of the access assembly (see <FIG>), and others gestures.

Like the example of a device-free gesture, in the example of the device gesture <NUM>, the access assembly <NUM> may not perform the motion detection or measurement. All such analysis may remain with the application <NUM> as part of the mobile device <NUM>. Optionally, the mobile device <NUM> may include the RSSI module <NUM> which can measure periodic variation signal strength of a beacon signal <NUM> as a result of the mobile device <NUM>, repetitiously, moving across the beacon signal path, or wireless interface <NUM>.

Referring to <FIG> and <FIG>, the gesture-based access control system <NUM>, in one embodiment, may be a knocking gesture access control system. In this embodiment, the user <NUM> of the mobile device <NUM> performs a knock that may be a predefined frequency of knocks. The term "knock" in the present embodiment would include the act of tapping. The knocking may be performed on the mobile device <NUM>, the access assembly <NUM>, the door <NUM> (see <FIG>), a wall area proximate to the access assembly <NUM> and/or door <NUM>, or any other surface conveniently located near the access point.

The mobile device <NUM> of the knocking gesture access control system <NUM> may further include a microphone <NUM>, and a knock module <NUM> of the application <NUM>. The microphone <NUM> may be sensitive enough to detect a wide range of frequencies and magnitudes (i.e., loudness) to track the sound originated by repetitious knocking on, for example, a surface (e.g., front surface) of the mobile device <NUM>, a surface of the door <NUM>, a surface of the door frame <NUM>, a surface of the access device <NUM>, a surface of a wall <NUM> through which the door <NUM> provides access, or other surfaces. The knocking is an intentional gesture performed by the user <NUM> (see knocking gesture <NUM> in <FIG>. Knocking or tapping on the mobile device <NUM> may be considered to be a device gesture as a type of intentional gesture, and knocking on any other surface may be considered to be a device-free gesture as a type of intentional gesture.

In one embodiment, the knock module <NUM> of the application <NUM> is configured to receive the signature of, or information relative to, the audible sound created by the knocking gesture <NUM>. The knock module <NUM> may then compare a measured frequency pattern of the audible sound (i.e., frequency of knocks or taps) to a preprogrammed frequency pattern. In one embodiment, if the measured frequency pattern sufficiently compares to, or substantially matches, the preprogrammed frequency pattern, the knock module <NUM> may determine that the knocking gesture <NUM> was performed by the user <NUM>, and effect the sending of the command signal <NUM> to the access assembly <NUM>.

In another embodiment, the knocking gesture access control system <NUM> may be configured to further confirm (e.g., independently confirm) performance of the knocking gesture to enhance reliability and reduce or eliminate false gesture confirmations. One such confirmation may include use of the IMU sensor system <NUM> similar to that previously described. For example, if the mobile device <NUM> is in a back pocket <NUM> (see <FIG>) and the user <NUM> performs the knocking gesture <NUM> upon the door <NUM>, the mobile device <NUM> may still measure a motion (i.e., of the mobile device) attributable to the act of knocking. In certain scenarios (e.g., user walking), the actual motion measured may be a compound motion, and the application <NUM> is configured to decipher multiple motions from the compound motion. Once deciphered, the frequency pattern of the motion attributable by the knocking is compared to a preprogrammed motion frequency pattern (i.e., may be the same as the audible frequency pattern), if the motion frequency pattern compares to, or substantially matches, the preprogrammed frequency pattern, the confirmation that the knocking gesture was performed is re-affirmed.

In another embodiment, the knocking gesture access control system <NUM> may use other sensory data to re-affirm gesture confirmation. For example, light sensor data from the environment detecting system <NUM> and/or RSSI data produced by fluctuations of the beacon signal <NUM> and produced by the RSSI module <NUM> as previously described. In one embodiment, the knocking gesture <NUM> may be a device-free gesture. In this example and if the IMU sensing system <NUM> is applied, the location of the mobile device <NUM> may also be determined in ways previously described. The detection process applied to detect the knocking gesture <NUM> may fuse the various methods described and optionally, the mobile device location method, to provide good intent markers as part of the application <NUM>.

Referring to <FIG>, <FIG>, and in another embodiment, the knocking gesture <NUM> may be performed upon a front surface <NUM> of the mobile device <NUM>. The mobile device <NUM> is associated with the X-Y-Z coordinates illustrated in <FIG>. If the knocking gesture <NUM> is performed against the surface <NUM>, the audible knocking sound is evaluated as previously described. The re-confirmation of the detection utilizing the IMU sensing system <NUM> and conducted by the knock module <NUM>, may evaluate the motion along the Z-axis only to mask-off motion noise produced along other coordinates. That is, the knocking is performed against the front surface <NUM>, and the direction of the knocking is substantially normal to the front surface <NUM>.

It is understood and contemplated that the knocking on the mobile device <NUM> instead of the door <NUM> may prevent disturbing a person on the other side of the door <NUM>, where access is intended by the user <NUM>. It is further understood, that preconditions may apply before the knocking gesture <NUM> is accepted. Such a pre-condition may be a requirement that the user <NUM> is within a pre-defined proximity of the access assembly <NUM>, or door <NUM>. Moreover, the knocking on the mobile device <NUM> can be done before the uses <NUM> reaches the door. In contrast, the example of knocking on the door is when the user <NUM> has already arrived. Therefore, in the example of knocking on the mobile device <NUM> enables the user <NUM> to perform an action as the user walks up to the door <NUM>. The door <NUM> may then be unlocked when the user <NUM> arrives.

Referring to <FIG>, the gesture-based access control system <NUM> may be flexible and capable of automatically adjusting for different intentional gestures including the device gesture <NUM> (see <FIG>) and the device-free gesture <NUM> (see <FIG>). In addition, the access control system <NUM> may adjust for the array of motions (i.e., compound motions), locations, and positions of the mobile device <NUM> when determining if an intentional gesture <NUM>, <NUM> is being performed by the user <NUM>.

<FIG> illustrates a non-limiting plurality of mobile device <NUM> locations and uses, wherein the application <NUM> is capable of adapting to in order to determine if an intentional gesture <NUM>, <NUM> is being performed. Accordingly, with the determination of mobile device motion, location, position, and/or usage, the application <NUM> may be further capable of selecting an appropriate preprogrammed gesture from a plurality of preprogrammed gestures.

As previously described, the inertial measurement unit (IMU) sensor system <NUM>, the environment detection system <NUM>, and the internal activity notification module <NUM>, together, are capable of providing information used by the application <NUM> to determine if an intentional gesture <NUM>, <NUM> is being performed.

Examples of the potential multitude of mobile device <NUM> locations, positions, and uses are illustrated in <FIG> and may include depiction <NUM> representative of the mobile device <NUM> located at an ear <NUM> of the user <NUM> with a usage of conversing or calling, and a substantially vertical position. Depiction <NUM> represents the mobile device <NUM> being in a front shirt pocket <NUM> thus having a substantially vertical position and in a relatively dark environment. Depiction <NUM> is representative of the mobile device <NUM> in the hand <NUM> of the user <NUM>, positioned at about thirty degrees for texting, and with a usage of texting. Depiction <NUM> is representative of the mobile device <NUM> being in a front pants pocket <NUM>, thus having a substantially vertical position and being in a relatively dark environment. Depiction <NUM> is representative of the mobile device <NUM> being located in the rear pants pocket <NUM> (also see <FIG>) thus having a substantially vertical position and being in a relatively dark environment. Depiction <NUM> is representative of the mobile device <NUM> hanging. For example, the user <NUM> may simply be carrying the mobile device <NUM> in the hand <NUM>. Depiction <NUM> is of the mobile device <NUM> in a handbag (i.e., containment <NUM>, also see <FIG>), thus in a dark environment, and depiction <NUM> is of the mobile device <NUM> in a backpack (i.e., containment <NUM>, also see <FIG>).

Referring to <FIG>, the application <NUM> of the access control system <NUM> may include the activity notification module <NUM>, an environment module <NUM>, a motion module <NUM>, a selection module <NUM>, and a plurality of mode modules (i.e., five illustrated as 328A, 328B, 328C, 328D, 328E). The activity notification module <NUM> is configured to determine and/or categorized current usage of the mobile device <NUM>. Examples of usage include texting, conversing, standby, and others. The environment module <NUM> is configured to receive and categorize environment information (see arrow <NUM>) from the environment detection system <NUM>. As previously described, environment information <NUM> may include light level data, temperature data, position data, location data, photographic data, sound data, and other data. The motion module <NUM> is configured to receive and categorize motion information (see arrow <NUM>) from the IMU sensor system <NUM>. Non-limiting examples of motion information include the compound motion previously describe, and which may occur in a variety of scenarios including when the user <NUM> is walking, standing still, carrying the containment <NUM>, performing a usage, and a wide variety of other events that may produce motion. One or more of the modules <NUM>, <NUM>, <NUM> may include algorithms, which may be self-learning algorithms, and preprogrammed data (i.e., portions of the scenario data <NUM>) to refine and/or categorize the information <NUM>, <NUM>, and other data for use by the selection module <NUM>.

The selection module <NUM> is configured to apply the information outputs from the modules <NUM>, <NUM>, <NUM> and thereby select one of the mode modules <NUM>. In one embodiment, each of the mode modules <NUM> may be, at least in-part, associated with a respective depiction <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. The selection module <NUM> may include a preprogrammed matrix of data <NUM> and algorithm(s). The preprogrammed matrix of data <NUM> may be representative of the motion and parameter (i.e., environment and usage) data received from the modules <NUM>, <NUM>, <NUM>. At least from the matrix of data <NUM>, the selection module is capable of selecting the appropriate mode module <NUM>. This selection may occur prior to, or during, the performance of an intentional gesture <NUM>, <NUM>.

Each mode module 328A, 328B, 328C, 328D, 328E may include a respective, preprogrammed, scenario data 66A, 66B, 66C, 66D, 66E of the scenario data <NUM> previously. Each of the plurality of mode modules <NUM> may also include a respective one of a suite of intent detection algorithms <NUM> (i.e., see 336A, 336B, 336C, 336D, 336E) for each respective mode module illustrated. In operation, the selection module <NUM> is configured to generally activate the appropriate algorithm 336A, 336B, 336C, 336D, 336E by selecting the appropriate module 328A, 328B, 328C, 328D, 328E. Each algorithm 336A, 336B, 336C, 336D, 336E is characterized in accordance with the context where it is applied. For example, algorithm 336A may be suitable when the user <NUM> has the mobile device <NUM> in the hand <NUM>, but may be less suitable when the mobile device <NUM> is in the rear pants pocket <NUM>. Therefore, different mode modules <NUM> are enabled and disabled in real time by the selection module <NUM>.

In operation, when the appropriate, selected, mode module <NUM> conditionally detects the intentional gesture <NUM>, <NUM>, the mode module may output the command signal <NUM> to the access assembly <NUM>.

Referring to <FIG> and <FIG>, and in one embodiment, the gesture-based access control system <NUM> may be a seamless access control system adapted to allow access to a user <NUM> after the user provides an inherent gesture <NUM> (see <FIG>) signifying the intentional desire and initial act of, for example, opening the door <NUM>. More specifically, the inherent gesture <NUM> is the initial part of a typical user exercise <NUM> conducted to gain entry.

The mobile device <NUM> for the seamless access control system <NUM> may be a wearable mobile device. Examples of the wearable mobile device <NUM> include a smart watch, smart glasses, and smart shoe(s). The term "smart" is meant to indicate that the wearable mobile device <NUM> includes the processor <NUM> and other features/components previously described.

The access assembly <NUM> may further include a near field communication (NFC) device <NUM> for generating the beacon signal <NUM>. In one example, the NFC device <NUM> may be a Bluetooth device, the beacon signal <NUM> is a Bluetooth signal, and the wearable mobile device <NUM> is configured to process the Bluetooth signal. In one example, the proximity sensor <NUM> of the environment detection system <NUM> may be used to measure the strength of the beacon signal <NUM>, and through this measurement, the application may determine the proximity of the wearable mobile device <NUM> to the access assembly <NUM>.

The mobile device <NUM> may further include a magnetometer <NUM> and a confirm ground truth module <NUM> as part of the application <NUM> (see <FIG>). The magnetometer <NUM> may be leveraged to confirm, for example, the grabbing of a handle <NUM> of the door <NUM> as part of the inherent gesture <NUM>. As best illustrated in <FIG>, the inherent gesture <NUM> portion of the user exercise <NUM> may be a sequential set of motions made by the user. The sequential set of motions may be dependent upon the type of wearable mobile device <NUM> and the type of entry desired.

For simplicity of explanation and understanding that this is only one, non-limiting, embodiment of an application, the entry type to be gained will be described as entry through a door <NUM> (see <FIG>). Also in the present embodiment, the type of mobile device <NUM> is the smartwatch. In this example, the inherent gesture <NUM> of the user exercise <NUM> may begin with, at block <NUM>, a deceleration of walking and/or stopping completely. At block <NUM>, the user <NUM> may lift the hand <NUM>, carrying the smartwatch <NUM> with the hand, in order to reach a handle <NUM> of the door <NUM>. At block <NUM>, the hand <NUM> may grab the handle <NUM> preparing to pull or push the door <NUM> open. This grabbing action of the inherent gesture <NUM> may be sensed by the magnetometer <NUM> of the wearable mobile device <NUM>.

In operation, and after the inherent gesture <NUM> is performed and confirmed by the application <NUM>, the wearable mobile device <NUM> sends the command signal <NUM> to the access assembly <NUM> to effect actuation from the no-access state to the access state, and as previously described. With the access assembly <NUM> in the access state, and at block <NUM>, the user <NUM> may complete the entry exercise <NUM> by pulling (see arrow <NUM>) the door <NUM> open.

The confirm ground truth module <NUM> (see <FIG>) of the application <NUM> is configured to receive information from the IMU sensing system <NUM> indicative of the pulling <NUM> that designates the final step of the entry exercise <NUM>. This confirmed pulling <NUM> may be verified by a preprogrammed confirmation pull which may be part of the scenario data <NUM> previously described. By confirming that the user <NUM> did indeed conduct the pulling <NUM>, the module <NUM> is able to further confirm an accurate determination of the inherent gesture. This confirmation may then be used to further improve the machine learning algorithm(s) <NUM> (see <FIG>) and/or other applied algorithms executed by the application <NUM>.

In the example of the wearable mobile device <NUM> being smart glasses, the smart glasses may be worn about the head of the user <NUM>, and parts of the inherent gesture <NUM> may include the user gaze when proximate to the access assembly <NUM>, and tilting of the head when approaching the handle <NUM> of the door <NUM>.

In the example of the wearable mobile device <NUM> being smart shoes, the smart shoes may be worn on the feet of the user <NUM>, and part of the inherent gesture <NUM> may include the tapping of a foot of the user <NUM>.

Referring to <FIG> and <FIG>, the gesture-based access control system <NUM> may be a prestaging, gesture-based access control system. In this embodiment, the mobile device <NUM> is configured to pre-stage itself prior to the user performing a device, or device-free, gesture (i.e., a primary gesture). That is, the system applies implicit behavior detection in combination with an explicit gesture from a plurality of gestures. The prestaging event, or process, may be, or may include the performance of an inherent gesture <NUM> (see <FIG>). After performance of the inherent gesture <NUM> by the user <NUM>, the user <NUM> needs to perform the primary gesture within a prescribed duration of time. One, non-limiting, example of the inherent gesture <NUM> may be the act of slowing down a walk as the user <NUM> approaches the access assembly <NUM>.

Referring to <FIG>, the application <NUM>, with any relevant hardware, may further include a timer or clock <NUM> and a satellite-based location module <NUM> (e.g., global positioning system (GPS). In another embodiment, the satellite-based location module <NUM> may be a separate device from the application <NUM>, which is configured to send pertinent location information to the application <NUM>.

In order to detect the prestaging event (i.e. inherent gesture <NUM>), the IMU sensing system <NUM> may be active. The activation of the IMU sensing system <NUM> may be triggered when the user <NUM> is within a prescribed vicinity of the access assembly <NUM>. Establishing a user <NUM> presence within the vicinity may be established in any one of a variety of ways. For example, any one or more of the following may be used: the satellite-based location module <NUM>, the proximity sensor <NUM> of the environment detecting system <NUM>, detection of the beacon signal <NUM> generated from the NFC device <NUM> of the access assembly <NUM>, and others.

In one, non-limiting, embodiment the implicit detection of an access intent of the user <NUM> may rely on the intuition that the user will slow down, and stop, as the user approaches a destination door <NUM> associated with the access assembly <NUM>, and perform a primary, intentional gesture, to indicate the intent. This intuition may be leveraged to improve the reliability of gesture detection.

Referring to <FIG>, a method of operating the prestaging, gesture-based, access control system <NUM> is illustrated. At block <NUM>, the IMU sensing system <NUM> is initiated, wherein the IMU analytics performed by the motion module <NUM> of the application <NUM> are started. At block <NUM>, the motion module <NUM> determines if, for example, the user <NUM> is walking. At block <NUM>, and if the user <NUM> is walking, the motion module <NUM> determines if the user <NUM> is slowing down the walk (i.e., the inherent gesture <NUM>). If the walking is slowing down, the inherent gesture <NUM> (in this example) is detected.

At block <NUM>, and after the user <NUM> is detected, or confirmed, via the inherent gesture <NUM>, the application <NUM> may start a timer <NUM> thereby running a prescribed time duration. At block <NUM>, and during the prescribed time duration, the mobile device <NUM> monitors for the occurrence of a primary, intentional, gesture. If the primary, intentional, gesture is detected and at block <NUM>, the application <NUM> effects the output of the command signal <NUM> to the access assembly <NUM> (e.g., open door <NUM>). It is contemplated and understood that the primary, intentional, gesture may be a device gesture, a device-free gesture, and/or another inherent gesture.

At block <NUM>, as an optional step, and if the primary intentional gesture has yet to be detected, the motion module <NUM> of the application (or by other means) may determine if the user <NUM> has, for example, stopped walking altogether. If no, the application <NUM> continues to monitor for the performance of the primary, intentional, gesture. This optional step may assist when the gesture detection is not at a high confidence level. If the user <NUM> has stopped walking and at block <NUM>, the application <NUM> determines if the time duration has expired. If the time duration has not expired, the application <NUM> continues to monitor for the performance of the primary, intentional, gesture. If the time duration has expired, the process is deactivated, or the motion module <NUM> is re-initiated for detection of the prestaging, inherent, gesture (i.e., prestaging event performed by the user <NUM>) if the user <NUM> remains in the vicinity of the access assembly <NUM>.

It is contemplated and understood, that at any stage during the process (e.g., at block <NUM>), the mobile device <NUM> may provide audible and/or visual notifications to the user <NUM>. For example, the mobile device <NUM> may notify the user <NUM> that the mobile device is waiting upon the performance of the primary, intentional, gesture. As another example and upon expiration of the time duration, the mobile device <NUM> may inform the user <NUM> that detection of the primary, intentional, gesture has failed.

In one embodiment, the prestaging event may be preprogrammed, and the primary intentional gesture may be pre-selected from a plurality of preprogramed gestures by the user <NUM>. Non-limiting examples of the primary, intentional, gesture may include: the waving of the hand <NUM> near the access assembly <NUM> (i.e., a type of device-free or body gesture <NUM>, see <FIG>); tapping on the door <NUM> or the access assembly <NUM> (a type of device-free or body gesture <NUM>, see <FIG>); a specific body gesture triggering inertial motion, wherein the mobile device is attached to the body of the user (also see <FIG>); applying a body motion to a containment <NUM> containing the mobile device <NUM> and carried by the user <NUM> (i.e., a containment motion <NUM>, see <FIG>); the waving of the mobile device <NUM> near the access assembly <NUM> (i.e., a type of device gesture <NUM>, see <FIG>).

Referring to <FIG>, the gesture-based access control system <NUM> may include use of a cloud <NUM> (i.e., remote server). In this embodiment, the application <NUM> may be in the cloud <NUM>, thus information <NUM>, <NUM> gathered by the IMU sensing system <NUM>, the environment detecting system <NUM>, and other components may be wirelessly sent from the mobile device <NUM> and to the cloud <NUM> for processing. The command signal <NUM> may be sent directly from the cloud <NUM> and to the access assembly <NUM>, or back to the mobile device <NUM> that then sends the signal <NUM> to the access assembly <NUM>.

Benefits of a cloud-based architecture include the performance of some or all computations and the storage of data in the cloud. This permits use of what may be more powerful algorithms, but at the potential expense of delay in communication. Another advantage may be that the mobile device <NUM> does not need to communicate directly with the access assembly <NUM>, and instead, the cloud <NUM> communicates a command signal directly to the access assembly <NUM> for access granting.

Advantages and benefits of the present disclosure include enablement of gesture detection without the need to hold a mobile device <NUM> in the hand. Another advantage includes the ability to identify, for example, a door <NUM> that a user <NUM> intends to enter as part of the intent detection. Yet other advantages include reliable intent detection, and a relatively inexpensive and robust design.

The various functions described above may be implemented or supported by a computer program that is formed from computer readable program codes, and that is embodied in a computer readable medium. Computer readable program codes may include source codes, object codes, executable codes, and others. Computer readable mediums may be any type of media capable of being accessed by a computer, and may include Read Only Memory (ROM), Random Access Memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or other non-transitory forms.

The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "includes," "including," "comprises," and/or "comprising," when used in this specification, specify the
presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

As used herein, the term "if' is, optionally, construed to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" is, optionally, construed to mean "upon determining" or "in response to determining" or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]," depending on the context.

Claim 1:
A method of training a software-based application (<NUM>) for a gesture-based access control system (<NUM>), comprising:
communicating first training instructions to a user (<NUM>) of a mobile device (<NUM>) via a human interface device (<NUM>) of the mobile device, wherein the training instructions instruct the user to perform a first motion with the mobile device at a first location on the user;
performing the first motion with the mobile device at the first location on the user; preprogramming scenario data (<NUM>) comprising a user specific model utilizing information received from at least one detection system (<NUM>, <NUM>, <NUM>, <NUM>) of the mobile device while the user performs the first motion, wherein the information includes motion data;
communicating second training instructions to the user via the human interface device, wherein the second training instructions instruct the user to perform the first motion with the mobile device at the first location on the user with a chosen intentional gesture (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) representative of an intent of the user to gain access to an access assembly (<NUM>);
performing the first motion with the mobile device at the first location on the user with the chosen intentional gesture;
preprogramming the scenario data (<NUM>) utilizing information received from the at least one detection system while the user performs the first motion with the chosen intentional gesture, wherein the information includes motion data; and
training, using the scenario data (<NUM>), a machine learning algorithm for gesture recognition,
wherein the machine learning algorithm is configured to determine or confirm an intent of the user based on an explicit intent signal generated by the at least one detection system, and to determine user authentication based on the intent signal generated by the at least one detection system and the user specific model.