Patent ID: 12242914

The figures depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

DETAILED DESCRIPTION

System Architecture

FIG.1is a block diagram of a system environment100in which a localization system155operates, in accordance with an embodiment. The system environment100depicted inFIG.1includes wireless tags105(representing an asset or a person to the localization system155), wireless sensors115(such as vibration or CO2 sensors and static wireless or IR beacons), and (in the physical environment) heat sources110(such as people) that can be detected by ALDs120(Augmented Localization Devices) or the wireless sensors115.

As described above, the wireless tags105can communicate with one or more access points125over a Bluetooth Low Energy (BLE) connection (or other low energy wireless connection). Tag105communications with an access point125can include packets in standardized beacon formats (such as iBeacon® or Eddystone®) and/or additional telemetry or tracking packets specific to the localization system155(collectively referred to herein as “tracking packets”). The system environment100can also include one or more augmented localization devices120ALDs, that can communicate with the access points125via BLE (or other low energy wireless connection) and/or wireless tags105using an IR (infrared) connection. As described above, an ALD120can include sensors and provide functions to improve the accuracy of tag105localization.

The access points125relay messages received from the wireless tag105(and RSSI data for the messages) to the localization system155either directly (through a LAN and/or WAN) or, in some implementations, (such as when one or more access points125are provided and/or maintained by a third party outside the localizations system) through an intermediate system associated with the access points125(labeled the access point vendor cloud130inFIG.1). For example, the access point vendor cloud130can receive tracking information through webhooks sent from access points125when communications are received from the wireless tags105. The access point vendor cloud130can then forward that information through to the localization system155using, for example, a Firehose API stream or other method of continuously providing received tracking information (such as, for example tracking packets, RSSI information, and/or sensor data) to the localization system155.

The localization system155represented inFIG.1includes the ingress point cloud135, a room fingerprinting subsystem140(in some embodiments), a location subsystem145, and an application cloud150. The subsystems of the localization system155can internally communicate using real-time data streams (for example, Kafka streams) or a shared database.

The ingress point cloud135of the localization system155can receive tracking data from the access points125(for example, including received communications from tags105, sensor data, and communications from one or more ALDs120) and prepare that data for analysis by the rest of the localization system155. Similarly, the ingress point cloud135can be responsible for managing the connected devices (including ALDs120and access points125) and can, for example, detect new devices or if one or more devices go offline, fail to respond to a query, or otherwise need maintenance.

The room fingerprinting subsystem140of the localization system155can receive room reference data from one or more ALDs120and generate or update a room fingerprint or room fingerprint model associated with an ALD120or room (or otherwise use room reference data to improve localization). In some implementations, the location subsystem145analyzes the received tracking data to determine a location for each tracked tag105. When determining a location for a tag105, the location subsystem145can also factor in other data, such as room fingerprints for one or more rooms in the tracked space or occupancy change data reflecting changes in occupancy for one or more rooms. The determined location for a tag105can be represented by a room ID for the room the tag105is predicted to be inside, another indication of a specific room (or set of rooms), and/or a set of coordinates of the tag105.

In some embodiments, the localization system155also includes an application cloud150which compares the determined tag105locations to one or more policies and/or takes actions based on the determined locations. The specific policies implemented by the application cloud150can depend on the context of the tracking (for example, what assets or people tags105are associated with and the type of building or space that is being tracked), but actions taken by the application cloud150can include notifying one or more parties, remotely controlling one or more devices inside or outside of the tracked space, logging the location data, and/or generating an application interface protocol, for example a stream, to communicate with one or more business applications and/or legacy systems155(for example, via an API stream or set of API calls).

The localization system155may also interface with business applications and/or legacy systems155which can make use of the location information determined by the localization system155. For example, some legacy system155may have previously relied on manually entered location data (or other, less accurate, methods of determining tag105location) and can be updated to use location information from the localization instead. Business applications and/or legacy systems155can receive simplified location data, such as a set of geospatial coordinates or a room ID number and identifier of the tracked space (such as a building address).

FIG.2is a block diagram of a wireless tag105, according to an embodiment. As described above, a wireless tag105(e.g., tag105) can be a battery-powered device whose location can be tracked by the localization system155. Each tag105includes a CPU205RF transceiver235(for example, to send or receive BLE communications), antenna220, and battery215(or other power source). In addition, a tag105can include other components to provide additional functionality, such as a temperature sensor, IR receiver210, a user interface (UI)225, and/or a motion sensor230.

To be tracked by the localization system155, a tag105can broadcast tracking packets (over BLE) including the MAC address (Media Access Control address), or other identifier of the tag105. Depending on the implementation, a tag105can broadcast tracking packets periodically or based on the detection of one or more events. For example, a tag105can broadcast a tracking packet every minute, can broadcast in response to a UI input, based on receiving an IR communication, or based on detecting motion of the tag105.

The tracking packets broadcast by the tag105can include additional tracking information that can help refine the location of a tag105. For example, the IR receiver can be used, for example, to receive room ID codes broadcast from one or more ALD120sor beacons. These room ID codes can then be included in a corresponding tracking packet broadcast by the tag105. Similarly, motion sensor230data, time and date, and/or information received or derived by communication with an ALD120or beacon300can be incorporated into tracking packets sent by a tag105.

The CPU205of the wireless tag105acts as a controller. The CPU205, for example, may receive data from an IR or RF transmission and record a timestamp and/or RSSI associated with the transmission. The CPU205may additionally perform an action based on the content of received signals, such as making a change to the UI225, activating the motion sensor230, or sending a responsive signal, etc.

FIG.3is a block diagram of a beacon300, according to an embodiment. A beacon300is a static (or mostly static) device whose location is known to the localization system155. A beacon can include a CPU305, RF transceiver335(for example, to send or receive BLE communications), an antenna320, and battery315(or other power source). In addition, a beacon300can include other components to provide additional functionality, such as a temperature sensor and/or IR transmitter310. ALDs120, access points125, and other devices may be examples of beacons300and a tracked space can additionally include standalone beacons300(for example, IR beacons to transmit an IR room ID code). In some implementations, beacons300periodically transmit BLE packets or other low energy RF packets to surrounding devices.

As described above, augmented localization devices (ALD120s) are devices used by the localization system155to improve the accuracy and reliability of tag105localization. In some embodiments, the location of each ALD120is known to the localization system155and tracking data received from an ALD120can be linked to its location in the tracked space. Depending on the implementation, ALDs120can vary in included components and capabilities, with some being able to also function as access points125(or “gateways”) and others being standalone devices operated in addition to access points125and gateways. Some implementations of a localization system155can include ALDs120with different capabilities. For example, to retrofit an existing space for object location tracking, a relatively small set of ALD120gateways can be dispersed to key rooms to increase access point125coverage of the space (in addition to or replacing existing access points125) and a larger set of standalone ALDs120can be placed around the remainder of the space at a relatively lower cost (for example, one standalone ALD120to each room without a gateway). The CPU305of the beacon300acts as a controller.

FIG.4Ais a block diagram of a first implementation of an augmented localization device400A (an embodiment of the ALD120), according to an embodiment. The ALD400A ofFIG.4Aincludes a CPU405, RF transceiver410, corresponding antenna system415(such as an omnidirectional antenna and/or multiple antennae and a multiplexer), and a power source (such as a battery450, Power over Ethernet implementation, or DC power unit455). An ALD400A can include one transceiver410and one antenna415or, in other cases, multiple transceivers410with multiple antennae415or one transceiver410, a multiplexer, and multiple antennae (omni or directional). Using directional antennae (or other multi-antenna configurations) may increase the reference data available to the localization system155from the same source (for example, multiple IDs can be transmitted associated with a single device and a single reference location). Additionally, this ALD400A includes an occupancy sensor to detect the presence of one or more people near the ALD400A. For example, an occupancy sensor can include a thermal camera, motion detector, or the like. An occupancy sensor system may be capable of separating people clustered together and/or eliminating false positive heat signatures such as those generated by non-human sources (such as a projector or a hot mug of coffee). In some implementations, an ALD120also includes a neural network CPU to process data from the occupancy sensor (for example, to process thermal camera image data to detect people in the captured images). This ALD120also includes a UI for receiving input from and/or communicating with one or more users and a Wi Fi module that can allow the ALD120to function as a gateway. As described above, an ALD120can include an IR transmitter that can transmit a unique (or locally unique) room ID to tags105in the same room as the ALD400A. Similarly, the ALD400A can also function as a fixed RF beacon.

ALDs400A can also include a sensor array used to augment other IoT functions of the localization system155(or supported by the localization system155). For example, the ALD400A ofFIG.4Aincludes a sensor array445which may include temperature, humidity, carbon monoxide, air quality, light, and motion sensors. In some embodiments, an ALD400A additionally includes acoustic sensors (such as omni or directional microphones). These sensors can be used to implement one or more policies or features of the application cloud150. For example, the light sensors can be used to detect when lights are left on in a room that is currently unoccupied. Other implementations of an ALD400A can include other, more, or different sensors depending on the situation and desired IoT functionality.

FIG.4Bis a block diagram of a second implementation of an augmented localization device400B, according to an embodiment. WhileFIG.4Adepicted an ALD400A gateway,FIG.4Bshows an example of a standalone ALD400B. The standalone ALD400B ofFIG.4Bincludes a CPU405, RF transceiver410, antenna420, IR transmitter430, UI435, and power source (here, a battery425). In this case the standalone ALD400B includes a directional antenna system420using a multiplexer415to receive and transmit BLE signals. In some implementations, standalone ALDs400B that are battery powered can disable continuous usage of their RF transceivers410for receiving BLE signals (for example from tag105) for power conservation purposes. Instead, standalone ALDs400B can rely on other methods such as IR transmitted room codes, beacon transmissions over BLE, and/or limited-time use of the receiving capabilities of the RF transceiver to develop a room fingerprint based on RSSI data received at the standalone ALD400B. To enable an ALD400B to function on battery power for an extended period of time, a battery powered ALD400B may communicate with the access points125primarily (or exclusively) using the low energy network (for example, BLE). Implementations using, for example, Wi Fi networks for data backhaul may draw too much power to operate on battery425alone (and require expensive power cabling or other additional infrastructure support). The RF transceiver410may be turned off to conserve power, for example, if the device has not detected any RF signals for an interval of time (e.g., 5 minutes or 30 minutes).

FIG.5is a block diagram of the room fingerprinting subsystem140of the localization system155, according to an embodiment. Room fingerprint models can be machine learning models which create, update, and/or generate room fingerprints for one or more rooms within a tracked space based on room reference data. In some embodiments, the room fingerprint for a room predicts expected sensor data for a device located within the corresponding room (e.g., a range of expected RSSI values of a BLE source attempting to send or receive tracking packets from within the room). Room fingerprints may include time domain information (for example, to reflect expected changes in the room fingerprint over the course of a day as the RF environment of a room evolves).

In some implementations, the room fingerprinting subsystem140includes a machine learning module510which can train room fingerprint models. In some implementations, room fingerprint models can be trained based on room reference data including RSSI data that the ALD120of each room periodically gathers from transmissions of fixed BLE sources (for example, other ALDs120in adjacent rooms, access points125, or beacons) surrounding the room. Room fingerprint models can be dynamically updated (for example, daily) to improve the quality of the room fingerprint and because changes in the tracked space (for example, moving furniture or other large objects) can affect the room fingerprint for a room. In some implementations, a room fingerprint model is a machine learning process which creates, records, and updates a room fingerprint represent the room and modeling the environment's impact on signals traveling into and out of the room. A room fingerprint can be associated with a room ID and includes a range of RSSI values which are expected when sending and receiving signals from or into the room.

The room fingerprinting subsystem140can also include a room fingerprint database520storing the generated room fingerprints for access by the localization system155. As described above, the subsystems of the localization systems155can communicate using Kafka data streams (in the case of the room fingerprinting subsystem140, to receive the RSSI data needed to generate or update the room fingerprints), and API functions (for example, to provide room fingerprints from the room fingerprint database to the location subsystem145or a shared database).

FIG.6is a block diagram of the location computation subsystem145of an example localization system155, according to an embodiment. The location subsystem145can include a location module610to perform localization calculations (for example, trilateration) based on received tracking data as well as other modules that can prepare tracking data to be factored into the localization calculation. For example, a trilateration calculation may yield multiple possible results, for example by using a particle filter (Markov chain), and probabilistically select the “best” outcome. Localization augmentation data (like occupancy data) can be used to adjust the relative weights of each possible outcome. For example, if one possible room has a person in it and the other candidate rooms are empty (and the localization system155is attempting to locate a tag105identified as attached to a human), the occupied room can be assigned a higher relative weight.

For example, the location subsystem145can include a sliding time window data cache630that temporarily stores tracking data as it is received from access points125. Localization calculations can be based on data from different sources (including directly received data and data received through a vendor or 3rdparty system) that may result in a delay in receiving tracking data related to the same tag105broadcast.

The room reference module620can retrieve an appropriate room fingerprint from the room fingerprinting subsystem140that can be factored into the localization calculation. Similarly, the occupancy module640can receive and supply occupancy data for rooms in the tracked space (for example, as determined by ALD120s) to the location module. Similarly, the device profile database650can store information about each tracked tag105and/or access point125that can affect the calculation (such as an expected format of the tracking packet, a baseline signal strength for the transmission of the tracking packet, the nature of the object to which the tag105is attached to, and the like). In other embodiments, the location subsystem145may include additional, fewer, or different components for various applications. Conventional components such as network interfaces, security functions, load balancers, failover servers, management and network operations consoles, and the like are not shown so as to not obscure the details of the system architecture.

FIGS.7-13illustrate use cases of the localization system155. In these figures some objects are referred to with reference numbers followed by a letter (e.g., access point125a). In cases where the object is followed by only the number and no letter (e.g., access point125), the text may describe all devices of that kind present in the figure (e.g., access points125a,125b,and125c) or any one of such devices, depending on the context. Note that, for example, access points125a,125b,and125cmay be independent devices rather than a single device that has moved.

FIG.7illustrates an example tracked space in which a localization system155tracks a tag105, according to an embodiment. The room highlighted in the example tracked space ofFIG.7includes multiple wireless tags105aand105bincluding one with an IR receiver, an ALD120, and a heat source110(e.g., a human). Outside of the room, but still within the same building, are several access points125with a network connection to the localization system155(for example, taking BLE signals and relaying them using TCPIP over an 802.11 LAN or Ethernet connection). Other rooms in the tracked space include fixed BLE sources such as ALDs120positioned in other rooms and other BLE beacons, such as beacon300.

FIG.8illustrates localizing a wireless tag105in an example tracked space, according to an embodiment. To begin the process, the wireless tag105transmits a BLE packet (could be an iBeacon®, Eddystone®, Telemetry, or other tracking packet) including a MAC address of the tag105to an access point125(any or multiple of a, b, or c) or gateway. ALD120gateways or access points125receive the tracking packet and capture the receiving time and the received RSSI of the tracking packet. Each access point125transmits this tracking data (for example, tracking data can include a MAC address or other identifier of the tag105, the RSSI, and a time the tracking packet was received) to the localization system155(for example through the internet). The localization system155uses the tracking data for the tag105to calculate a location baseline for the tag105with or without interpolating other sources.

In the shown example, each access point125a,125b,and125cmay receive the same BLE tracking packet from the wireless tag105but receive it with a different RSSI due to their positioning relative to the tag105. For example, there may be an open door between the wireless tag105and access point125a,allowing access point125to receive the tracking packet with a higher signal strength than access points125band125cwhich may be separated from the tag105by signal attenuating walls. The localization system155may include a model of the building and its rooms with a detailed schematic of walls, doors, and windows. This model allows the localization system155to determine the location of the wireless tag by interpreting the signal strengths from each access point125based on the position of the access points. Therefore, though access point125areceives the highest RSSI in this example, the localization system155determines that the wireless tag105is actually closest to access point125cbut is separated by a wall.

As described above, the localization system155can factor in room occupancy data when trying to localize tags105to particular rooms. For example, a tag105can correspond to a person or other moveable asset (such as an infusion pump, which can only change its location when moved by a human).FIG.9illustrates using an augmented localization device to determine an occupancy model of the tracked space, according to an embodiment.

In some implementations, tracking occupancy can begin with the ALD120aactivating a thermal camera (triggered, for example, every ten second, every, minute, or based on detected motion, etc.). Using machine learning, the ALD120acan analyze the thermal camera output to determine a number of people in the room (based on their IR heat signatures) and determine coordinates of those people relative to the ALD120a.The ALD120acan then transmit the determined occupancy data (for example, number of people and coordinates) to a nearby access point125using BLE. Any of access point125a,125b,or125cmay be used. In some cases, all 3 access points may receive the occupancy data. The access point125can then forward the occupancy data to the localization system155, which uses the new occupancy data to update an occupancy module of the tracked space. The localization system155can use occupancy data and (including occupancy change data when rooms are entered and left) to inform a location probability model and increase tracking accuracy and/or for other IoT features. The localization system155can use this data to inform functionality relying on occupancy or occupancy changes, for example to understand space utilization or to verify that a space is empty in emergency situation (i.e., evacuated) or to determine human traffic and journey times inside buildings. For example, detecting if a shared use room (such as a conference room or shared workspace) is occupied.

For example, if the process ofFIG.8was used, the RSSI data from the access points125may not be enough for the localization system155to determine if a tag105(e.g., worn by a human) is in a first room or a second adjacent room (not shown). The occupancy data showing a heat source110in the first room confirms that the tag105associated with the heat source110is in the first room. In the case of multiple tags105and/or multiple heat signatures the occupancy data may be used by the localization system to determine weighted possibilities of the tag105locations.

In some implementations, the localization system155can implement IR augmented localization, using wireless tags105with IR receivers and ALD120sconfigured to broadcast coded IR signals.FIG.10illustrates localizing a tag105using an IR beacon, according to an embodiment. Wireless tags105with IR receivers can periodically wake up and record an encoded RoomID transmitted by the ALD120of the current room. Waking up may entail the tag105going from a low powered sleep mode to a fully active mode. Alternatively, the tag105may wake up upon detecting an IR signal and record that signal. The tag105may also be programmed to wake up and enter sleep mode periodically or after a set interval of time wherein no signals have been received. Then the tag105can broadcast a tracking packet including the received RoomID along with a tag identifier such as the MAC address of the tag105to nearby access points125. The RoomID may be an identifier that is unique or locally unique (such as within a building or within a single floor of a building) to the room. The receiving access point125can relay the received tracking packet to the localization system155as discussed above.

Because IR signals do not penetrate walls or other solid objects well (compared to, for example BLE signals), the localization system155can infer that tags105receiving an IR signal with and RoomID from an ALD120ashare a room with that ALD120a.However, because the bandwidth of the coded IR signals transmitted from an ALD120may be severely limited (sometimes to only a few digits), a tracked space may have too many rooms to assign each room or ALD120a unique RoomID.

To circumvent this issue, rooms can be assigned locally-unique RoomIDs (RoomIDs for which no duplicates exist in an immediate vicinity, even if a duplicate RoomID is assigned somewhere else in the building). When processing locally-unique RoomID, the localization system155can perform an RoomID duplication removal method to extract a single RoomID and/or coordinate of the tag105based on the received RoomID. For example, the duplication removal process can use an estimated location of the tag105(without the benefit of the RoomID data) and look up the nearest room associated with a matching RoomID. The localization system155can then use the IR data to improve location computation of the tag105(based on RSSI or other data sources). In some implementation, a matched RoomID can override other location calculations.

Similarly, a localization system155can use ALD120sto determine room fingerprints to aid in the localization of tag105s.FIG.11illustrates determining a room fingerprint using an augmented location device120, according to an embodiment. A room fingerprint is characterized by what objects (access points, beacons, tags, etc.) an ALD in a room is able to communicate with. To develop a room fingerprint for a room an ALD120ain the room opens a radio receiver and records BLE broadcasts from fixed BLE sources (access point125a,125b,and125c,ALDs120band120c,and beacon300). In some implementations, this process occurs periodically (for example, once per day for one minute). The ALD120acan record RSSI information, a source MAC address (or other identifiers), and/or Time of Flight information for BLE packets received from the fixed BLE sources and/or other sources in the tracked space. For example, objects such as other ALD120s,BLE beacons, and access point125sare considered fixed and can be used to develop a room fingerprint. The ALD120can then deliver room reference data that is based on the received BLE packets over BLE to a gateway or access point125b.The access point125brelays the room reference data to the localization system155. As described above, the localization system155can use the room reference data to feed a ML based fingerprint store. The fingerprint database can be dynamically updated with reference RSSI levels from within the room (as captured each day) or to update a statistical model of RSSI levels within a room. Similarly, to the occupancy data, the localization system155can create a model of the objects in the room based on the room reference data and use that model to weight predictions of where tags105are located.

FIG.12illustrates augmenting localization information based on time of arrival (ToA) or RSSI information as received by a tag105, according to an embodiment. In some implementations, a tag105can also gather RSSI data (or other location data) from ALD120sand access points125to improve its own localization accuracy. In some implementations, ALD120s,access point125s,and/or BLE beacons300can transmit a reference BLE signal (typically using iBeacon® or Tracking packet format). Wireless tags105can periodically open their RF receivers to gather this information. In some implementations, the tag105filters received messages by device ID to only collect data from ALD120s,known BLE beacons, access points125, or other known BLE sources) and records time and received RSSI from the filtered devices. Additionally, a tag105(or localization system155) can implement Time of Flight (or Time of Arrival) methods to estimate range from the tag105to the ALD120sor other known sources. When the tag105sends out its tracking packet over BLE, the tag105includes the gathered RSSI and/or Time of Flight information in the transmission (and ultimately forwarded to the localization system155). The localization system155can then use the additional RSSI data gathered by the tag105to improve the accuracy of the corresponding location. For example, the localization system155can determine which other device (such as an ALD120or access point125) is closest to the tag105based on the tag's105gathered RSSI data and provide a higher weight to the room the closest device to the tag105is installed in when performing the localization calculations.

FIG.13illustrates augmenting localization using multiple disclosed methods, according to an embodiment. In some implementations, the localization system155employs some or all the techniques discussed above to improve the object location tracking performance using ALD120s.FIG.13discloses possible transmissions occurring in an environment when a localization system155is using multiple disclosed localization enhancement methods simultaneously. For example, a localization system155may improve the location accuracy of the same tracking packet using occupancy data, a RoomID included in the tracking packet, and room fingerprints for rooms in the tracked space, as described above.

FIG.14is an interaction diagram illustrating an example process for localizing a tag105using RSSI data and room fingerprints, according to an embodiment. The process ofFIG.14can begin when a tag105broadcasts1410a BLE tracking packet including a MAC address of the tag105to surrounding access point125sand/or gateways. The access point125s/gateways each forward1420the MAC address of the tag105, RSSI for the received transmission, and a time the transmission was received to the localization system155. At the localization system155, the location module610can begin analyzing the received tracking data to determine a location of the tag105. In some implementations, the location module610determines a rough estimate of the location of the tag105(for example, based on trilateration without using room fingerprints) and, using that estimate, queries1430the fingerprint store for room IDs and room fingerprints for the rooms in the estimated area. After the fingerprint store responds1440with the requested information, the location module performs a location calculation (such as trilateration) using the room fingerprints to determine a room ID of the tag105's location. The location module610sends1450location information for the tag105(including the room ID, MAC address of the tag105, time, confidence of the predicted location, and the like) to the application cloud150. Similarly, the room ID determination can be used to update the room fingerprint model for the room and generate an updated room fingerprint associated with the room ID.

FIG.15is an interaction diagram illustrating an example process for localizing a tag105using RSSI data and occupancy data, according to an embodiment. The process ofFIG.15begins similarly to the process ofFIG.14, but the location module additionally receives1520an identifier of a tag105such as a MAC, as well as an RSSI, timestamps, and/or ToF. The location module610asks1530the room fingerprinting subsystem150for a RoomID and fingerprint data of the surrounding area of the room. The room fingerprinting subsystem150responds1540to the location module610with the requested data. The location module610requests1550RoomID occupancy data from the occupancy module for a room associated with the RoomID (for example, after requesting occupancy data for rooms near the rough estimate of the location of the tag105) to factor into the trilateration calculation, and the occupancy module640returns1560the requested occupancy data of the room. For example, the location module can use the occupancy or occupancy change data to weight rooms with occupants as more likely locations for a tag105associated with a person or to prioritize rooms where an occupancy change has occurred as locations for tag105swhich have changed locations recently. In some implementations, occupancy data (for example a room ID and number of occupants) can also be used by the application cloud150(and other services) to trigger actions within the tracked space, without the need for specific tracking of tag105s.For example, occupancy data can be used to control lights or the thermostat for specific rooms, to determine space utilization for a desk or space, or determining if a space is empty in emergency evacuation scenarios. Once the location module610has the occupancy data it sends1570the data to the application cloud150. In some embodiments the application cloud150may respond with an action to perform based on the received data. The determined locations of the tags are sent to the room fingerprinting subsystem140to update1580the ML model.

FIG.16is an interaction diagram illustrating an example process for localizing a tag105using RSSI data and RoomID codes, according to an embodiment. In order to use RoomID information, a tag105can open an IR receiver and receive1610a broadcasted RoomID associated with its current room (for example, from an ALD120or other IR beacon). Wireless tags105can also automatically wake up when an IR receiver receives an IR transmission. When the tag105sends1620a BLE tracking packet, the recently received RoomID can be included (alongside the MAC of the tag105etc.). Access points125and gateways can similarly pass1630along the RoomID information to the localization system155. As described above, bandwidth limitations of the IR beacon (such as beacon300) can mean that the same RoomID is associated with multiple RoomIDs within a tracked space. To resolve these duplicates, the location module610determines an approximate of the location of the tag105and selects a room ID associated with the RoomID based on the approximate room location (for example, selecting the room associated with the correct RoomID closest to the approximate room location). In some implementations, room fingerprints are used to determine the approximate room location (and the RoomID is used to confirm or override the approximate results). After the lag is localized based on the RoomID, the tag105location can be used similar to the process ofFIG.14(for example, for updating ML models and/or forwarding to the application cloud150for further use).

FIG.17is an interaction diagram illustrating an example process for implementing an augmented localization device “listening window” for collecting room reference data to train room fingerprints, according to an embodiment. To collect room reference data, an ALD120can periodically open its RF receiver and receive1710BLE packets coming from fixed BLE sources (such as other ALDs120, access points125, etc.). In some implementations, this occurs for a fixed period each day (for example, one minute). On receiving BLE packets during the listening window, the ALD120records RSSI data (for example including an average and standard deviation of the RSSI), a MAC address or other ID of the fixed BLE source, and a time the transmission was captured). The ALD120can then forward1720this room reference data to the room fingerprinting subsystem140(for example, via an access point125). The room fingerprinting subsystem140, as described above can use the room reference data to train a ML model and update1730a room fingerprint for the room ID associated with the ALD120. The room fingerprinting database520can then be updated with the revised room fingerprint or for the ML model to self-improve.

FIG.18is an interaction diagram illustrating an example process for localizing a tag105using time of flight information between the tag105and a fixed BLE source, according to an embodiment. To get ToF data to a fixed BLE source, a tag105can listen for a broadcast from the fixed BLE source, the broadcast1810including time information giving an indication of when the BLE source will be listening for requests. The tag105can send a request1820for ToF during the listening slot, requesting ToF information and including a counter. On receiving the request the fixed BLE source responds1830with the ToF information. Time of flight information (for example, an ID of the fixed BLE source and an estimated range to the BLE source) can then be included in tracking packets transmitted1840by the tag105. The ToF information can be forwarded1850to the localization system155by the access point125sreceiving the tag's105tracking packet. The tracking packet may also include an identifier of the fixed BLE source, an identifier of the tag105, RSSI data, and other tracking information. In some embodiments, the localization system155can use the ToF information to improve that accuracy of the predicted location of the tag105.

FIG.19is an interaction diagram illustrating an example process for gathering RSSI data from nearby BLE sources using a tag105, according to an embodiment. To determine RSSI information for nearby BLE sources, a tag105can listen for broadcasts transmitted1910from the fixed BLE sources and record MAC addresses and RSSI information for each received broadcast. This information can then be included in tracking packets sent1920by the tag105to access points125and gateways. The access point125can forward1930the tag's105collected RSSI information to the localization system155. In some embodiments, the localization system155can use the tag105's collected RSSI information to improve that accuracy of the predicted location of the tag105.

In an implementation, a localization system155receives, from a plurality of access point125s,tracking information for a tag105within a tracked space comprising a set of defined rooms, the tracking information including a plurality instances of tracking data derived from the receipt of a tracking packet at an access point125, the tracking data including an identifier of the tag105, RSSI information of the wireless packet received at the access point125, and a time the tracking packet was received by the access point125. The localization system155can then determine an initial estimate of the location of the tag105within the tracked space and request room fingerprints for rooms nearby to the estimated location of the tag105, the room fingerprints comprising a range of expected RSSI values for a room. Then, based on the room fingerprints, the localization system155can perform location calculations (such as a trilateration operation or statistical filtering operation) to determine an improved estimate of the location of the tag105.

To gather room fingerprint data for a room, an ALD120fixed within the room can periodically listen for low energy RF signals from fixed sources outside the room, for example for one minute. The ALD120can record telemetry data for each fixed source received during this period, including an identifier of the fixed source, an average and standard deviation of RSSI for the source, and time(s) that transmissions from the source were received (or other statistical data about the reception of the tracking packet). In some implementations, the ALD120broadcasts the gathered telemetry information in one or more wireless tracking packets over the low energy RF network. One or more access points125in the space can receive the tracking packets and relay the telemetry information to the localization system155for use in determining a fingerprint for the room. Based on the telemetry information, the localization system155trains or updates a machine learning model that generates a room fingerprint for the room to improve the machine learning model. The generated room fingerprint can be stored for later use in localization.

CONCLUSION

The foregoing description of the embodiments has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the patent rights to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.

Some portions of this description describe the embodiments in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.

Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described.

Embodiments may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

Embodiments may also relate to a product that is produced by a computing process described herein. Such a product may comprise information resulting from a computing process, where the information is stored on a non-transitory, tangible computer readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the patent rights. It is therefore intended that the scope of the patent rights be limited not by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments is intended to be illustrative, but not limiting, of the scope of the patent rights, which is set forth in the following claims.