SYSTEM AND METHOD FOR PROVIDING LOCALIZATION USING INERTIAL SENSORS

A system and method for providing localization, including, during a training phase: obtaining a training dataset of accelerations, angular velocities, and known locations over time of vehicles moving in a defined area; and training a machine learning model to provide location estimation in the defined area based on the accelerations and angular velocities using the training dataset; and during runtime phase: obtaining runtime accelerations and angular velocities over time of a vehicle moving in the defined area; and using the trained model to obtain current location of the vehicle based on the runtime acceleration and angular velocities.

FIELD OF THE INVENTION

The present invention relates generally to localization technology and, more specifically, to localization based on inertial sensors.

BACKGROUND OF THE INVENTION

The need for high-accuracy localization, positioning and mapping solutions in real-time exists in many domains and applications. Current outdoor localization technology typically utilizes a satellite navigation device, also referred to as global navigation satellite system (GNSS) including for example, global positioning system (GPS), GLONASS, Galileo, Beidou and other satellite navigation systems. Drivers use GNSS systems routinely for localization and navigation. In addition, autonomous vehicle companies integrate localization and mapping sensors and algorithms to achieve high-accuracy localization solutions for driver safety.

However, GNSS cannot be used for indoor navigation, localization or positioning applications. Indoor navigation, localization or positioning applications may include, for example, navigating robots or vehicles in storage warehouses that are used to monitor and provide equipment efficiently, or navigating in an indoor parking lot. Today, indoor localization is typically performed by applying sensor fusion schemes, where data acquired by many types of sensors is integrated to provide an estimation of the location of the vehicle.

In addition, GNSS may not provide adequate accuracy for some outdoor localization applications as well. For example, localization systems for autonomous vehicles may require higher accuracy than is provided by GNSS. Thus, localization systems for autonomous vehicles may also use sensor fusion to achieve high-accuracy localization solutions. The sensors may include a camera, LIDAR, inertial sensors, and others. Unfortunately, these sensors may be expensive, and the quality of the data they provide may depend on various physical conditions, such as day and night, light and dark, urban canyon, and indoor environments. Hence, there is no high-accuracy localization and mapping solution for vehicles, both indoor and outdoor.

SUMMARY OF THE INVENTION

A computer-based system and method for providing localization may include: during a training phase: obtaining a training dataset of accelerations, angular velocities, and known locations over time of vehicles moving in a defined area; and training a machine learning model to provide location estimation in the defined area based on the accelerations and angular velocities using the training dataset; during runtime phase: obtaining runtime accelerations and angular velocities over time of a vehicle moving in the defined area; and using the trained model to obtain current location of the vehicle based on the runtime acceleration and angular velocities.

According to some embodiments of the invention, the accelerations, angular velocities of the training set and the runtime acceleration and angular velocities may be measured using at least one inertial measurement unit (IMU).

According to some embodiments of the invention, the IMU may include at least one three-dimensional accelerometer and at least one three-dimensional gyroscope.

According to some embodiments of the invention, the machine learning model may be a neural network.

Some embodiments of the invention may include, during the training phase: extracting features from the accelerations and angular velocities of the training dataset and adding the features to the training dataset; and during the runtime phase: extracting runtime features from the runtime accelerations and angular velocities; and using the trained model to obtain the current location of the vehicle based on the runtime acceleration, the runtime angular velocities and the runtime features.

According to some embodiments of the invention, the features may be selected from velocity and horizontal slope.

According to some embodiments of the invention, during the training phase, the known locations may be obtained from at least one of the list including: a global navigation satellite system (GNSS) receiver and a real-time kinematic (RTK) positioning system.

According to some embodiments of the invention, the defined area may include a route.

Some embodiments of the invention may include dividing mapping of the defined area into segments, and according to some embodiments, the location may be provided as a segment in which the vehicle is located.

Some embodiments of the invention may include performing anomaly detection to find changes in defined area.

Some embodiments of the invention may include obtaining readings from at least one sensor selected from, a GNSS receiver, a Lidar sensor and radio frequency (RF) sensor; and using the readings to enhance an accuracy of the current location provided by the trained ML model.

It will be appreciated that, for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn accurately or to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity, or several physical components may be included in one functional block or element. Reference numerals may be repeated among the figures to indicate corresponding or analogous elements.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components, modules, units and/or circuits have not been described in detail so as not to obscure the invention. For the sake of clarity, discussion of same or similar features or elements may not be repeated.

Today, vehicles, cars, robots, and other moving ground platforms, commonly referred to herein as vehicles, may use many sensors in a sensor-fusion framework to obtain a localization solution in real-time. The sensors used may include cameras, inertial sensors, Lidar, and RF sensors. These sensors typically suffer from considerable disadvantages and are unable to provide the high-accuracy needed for various scenarios, such as indoor localization where no GNSS reception is available and outdoor localization for autonomous vehicles where the accuracy provided by GNSS is still not high enough to allow safe-driving. Other applications that may require the high-accuracy navigation may include navigating around a parking lot and navigating a tractor or other agricultural vehicles in a field, where the tractor needs to cover an entire field efficiently, e.g., without leaving any part of the field uncovered and with minimal repetitions.

An inertial measurement unit (IMU) may be or may include an electronic device configured to measure the specific force, angular velocity, magnetic field and the orientation of a vehicle, typically using one or more accelerometers, e.g., three-dimensional accelerometers, gyroscopes, e.g., one three-dimensional gyroscopes, and optionally magnetometers. In some implementations, IMUs may be used in strapdown inertial navigation system (SINS), where the IMU sensor is physically attached to the body of the vehicle and measurements are integrated into motion equations. Moving along surfaces, roads, and other terrains results in a dynamic change of the IMU readings. As such, the sensor readings contain intrinsic knowledge regarding the changes in location, which may be used to calculate the current location of the vehicle. However, current IMUs used in SINS typically suffer from biases and drift over time, making SINS problematic for high-accuracy localization and positioning solutions when used alone, without any accurate measurement update.

Some embodiments of the invention aim to solve the high-accuracy localization and positioning problem in real-time for vehicles using inertial sensors by using machine learning (ML) models. For example, according to some embodiments of the invention, readings of the inertial sensors may be provided as input to a deep learning (DL) neural network (NN) model.

According to some embodiments of the invention, signals from IMUs may be provided to an ML model that may provide the position or location of the vehicle. The signals provided by IMUs may include information indicative of accelerations, angular velocities, and time as raw data. Additionally, features may be calculated based on raw data, such as velocity and horizontal slope, etc.

Some embodiments of the invention may include training a machine learning model to provide location estimation using a training dataset of accelerations, angular velocities, and known locations over time of vehicles moving in a defined area. By providing a large training dataset to the model and performing optimization techniques (e.g., training and testing using for example cross validation via k-folds), a functional mapping may be established. Once completed, raw data information measured by IMUs of vehicles, may be provided to the trained ML model, and the trained model may provide location or position estimation of the moving vehicle in real time.

According to some embodiments of the invention, the ML model may be or may include a NN model, and more specifically, a DL NN. A NN may include neurons and nodes organized into layers, with links between neurons transferring output between neurons. Aspects of a NN may be weighed, e.g., links may have weights, and training may involve adjusting weights. Aspects of a NN may include transfer functions, also referred to as nonlinear activation functions, e.g., an output of a node may be calculated using a transfer function. A NN may be executed and represented as formulas or relationships among nodes or neurons, such that the neurons, nodes or links are “virtual”, represented by software and formulas, where training or executing a NN is performed by for example a dedicated or conventional computer. A DL NN model may include many neurons and layers with non-linear activation functions such as convolutional, Softmax, rectified linear unit (ReLU), etc.

The training dataset may be generated in a tagging procedure for a given area or environment. For example, a designer may map the entire area (e.g., a route, a parking lot, tunnels, urban canyons and other areas where GNSS reception is poor etc.) where a batch of raw data may be tagged or labeled with the correct location. This process may be improved with user-recorded raw data information and shared on a cloud.

Thus, some embodiments of the invention may improve the technology of positioning and localization by providing high-accuracy localization solutions based on IMU signals and DL ML schemes in real-time. In addition, some embodiments of the invention may provide a database of terrain information for various areas such as routes, parking lots, tunnels and urban canyons and keep updating the database. Some embodiments may find anomalies in the received signals, adjust the DL NN model online, and provide notifications regarding terrain anomalies in a vehicle's network for safe driving, where pits and other danger road modification will be shared among all users in a defined zone.

FIG. 1depicts a system100for providing localization of a vehicle, according to some embodiments of the invention. According to one embodiment of the invention, system100may include a vehicle110, equipped with one or more sensor unit112that may measure and provide data including at least one of specific force, angular velocity and/or the orientation of a vehicle, typically using at least one accelerometer, three-dimensional accelerometer, gyroscope and/or three-dimensional gyroscope. For example, sensor unit112may be or may include an IMU or an SINS, e.g., may be physically attached to the body of vehicle110. Sensor unit112may further include a processor and a communication module for initial processing and transmitting of data measured by sensor unit112to navigation server130. In the example provided inFIG. 1, vehicle110may be a vehicle moving along a road, way, path or route120. This example is not limiting, and system100may include a vehicle moving in any defined area, such as a parking lot, a tunnel, a field, an urban canyon (e.g., areas is cities where reception of signals from GNSS is poor), or in another confined area, or an indoor vehicle, a robot or any other indoor vehicle moving in a confined indoor area. Sensor unit112may provide the data measured by sensor unit112to a navigation server130directly or through networks140.

Networks140may include any type of network or combination of networks available for supporting communication between sensor unit112and navigation server130. Networks340may include for example, a wired, wireless, fiber optic, cellular or any other type of connection, a local area network (LAN), a wide area network (WAN), the Internet and intranet networks, etc. Each of navigation server130and sensor unit112may be or may include a computing device, such as computing device700depicted inFIG. 7. One or more databases150may be or may include a storage device, such as storage device730. In some embodiments, navigation server130and database150may be implemented in a remote location, e.g., in a ‘cloud’ computing system.

According to some embodiments of the invention, navigation server130may store in database150data obtained from sensor unit112and other data such as ML model parameters, mapping of terrain and/or route120, computational results, and any other data required for providing localization or positioning data according to some embodiments of the invention. According to some embodiments of the invention, navigation server130may be configured to obtain, during a training phase, a training dataset of accelerations, angular velocities, and known locations over time of vehicles110moving in a defined area or route120, and to train an ML model, e.g., a DL NN model, to provide location estimation in the defined area or route120based on the accelerations and angular velocities using the training dataset. For example, navigation server130may be configured to obtain, during a training phase, a training dataset of accelerations, angular velocities over time as measured by sensor unit112. For generating the training data set, the data measured by sensor unit112may be tagged or labeled with the known locations. According to some embodiments, during the training phase, navigation server130may obtain the known locations from at least one of a GNSS receiver and a real-time kinematic (RTK) positioning system. Other methods may be used to obtain the location data.

According to some embodiments of the invention, navigation server130may be further configured to, during a runtime phase, obtain runtime accelerations and angular velocities over time of a vehicle110moving in the defined area or route120and use the trained model to obtain current location of vehicle110based on the runtime acceleration and angular velocities.

According to some embodiments of the invention, navigation server130may be further configured to, during the training phase, extract features from the accelerations and angular velocities of the training dataset and add the features to the training dataset. For example, the features may include velocity, horizontal slope and/or other features. Navigation server130may be further configured to, during the runtime phase, extract the same type of features from the runtime accelerations and angular velocities, and use the trained model to obtain the current location of the vehicle112based on the runtime acceleration, the runtime angular velocities and the runtime features.

According to some embodiments of the invention, navigation server130may have mapping of the defined area or route120. In some embodiments, navigation server130may divide the mapping of the defined area or route120into segments and may provide or express the location of vehicle110a segment in which the vehicle110is located. Referring toFIG. 2, a defined area or route120divided into segments is presented, according to some embodiments of the invention. InFIG. 2, the dashed squares/rectangles represent segments200, and the location of vehicle110is between segments210and220. Other methods may be used to provide or express the location of vehicle110, e.g., using coordinates.

FIG. 3shows a flowchart of a method according to some embodiments of the present invention. The operations ofFIG. 3may be performed by the systems described inFIGS. 1 and 7, but other systems may be used.

In operation310, a training dataset of accelerations, angular velocities, and known locations over time of vehicles moving in a defined area may be obtained. For example, the accelerations and angular velocities may be measured by a sensor unit112including, for example, an IMU, and the known locations may be obtained from a GNSS receiver and/or a RTK positioning system. Other positioning systems may be used. An example of raw data measured by sensor unit112is presented inFIG. 4, which depicts an example of accelerations in the x, y and z directions (labeled ax, ayand az, respectively) and angular velocities in the x, y and z directions (labeled wx, wyand wz, respectively), measured by a sensor unit112that is attached to a vehicle110moving along defined area or route120.

Defined area or route120may include a uniform or non-uniform terrain, where pits, speed bumps, and more artifacts may be presented. During the motion of vehicle110, sensor unit112may measure signals that may represent the movement of vehicle110, as sensor unit112may be physically attached or integrated with the body of vehicle110, and may record the raw data with respective time. Considering an indoor environment, location or position information may be obtained or generated manually or using any applicable indoor positioning methods including Wi-Fi positioning, capturing images of vehicle110over time and extracting the location or position of vehicle110from the images, etc. The position or location data may be provided in any applicable manner including spatial coordinates, segments, etc. For example, if the defined area120is a parking lot, the parking number may be used as the location indication or label. In some embodiments, the defined area or route120may be divided into segments, as demonstrated inFIG. 2, and location or position data may be provided as the segment in which vehicle110is present.

In operation320, features from the accelerations and angular velocities of the training dataset may be extracted and added to the training dataset. The features may include, for example, the estimated velocity and horizontal slope. The estimated velocity and horizontal slope may be extracted, calculated or obtained by applying classical approaches, such as integration of the integration of the accelerometer readings (e.g., the measured acceleration) and gyroscope readings (e.g., the measured orientation). The training dataset may include a plurality of recordings made by the same or different vehicles110moving again and again in the defined area or route120.

In operation330, an ML model, e.g., a DL NN or other model may be trained to provide location estimation in the defined area based on the accelerations and angular velocities (and/or extracted features) using the training dataset. For example, the training dataset may be provided to the ML model and used in the training phase to adjust model parameters (e.g., weights) of the ML model. For example, the model parameters may be adjusted through a backpropagation training scheme, while the parameters of the ML model may be tuned or adjusted over and over until a loss function is minimized. A generalization of the solution may be achieved by using a nonlinear activation function, such as Sigmoid, ReLU, and Softmax, and a large number of neurons, convolutional, and recurrent layers. Eventually, during the training phase (e.g., operations310-330), a trained ML model may be obtained or generated.

In operation340, runtime accelerations and angular velocities over time of vehicle110moving in defined area or route120may be obtained. As in the training phase, the accelerations and angular velocities may be measured by sensor unit112including, for example an IMU that is physically attached to the body of vehicle110. In operation350, runtime features may be extracted or calculated from the runtime accelerations and angular velocities, similarly to operation320. In operation360, the trained model may be used to obtain current location of vehicle110based on the runtime acceleration and angular velocities and/or features. For example, the dataset of accelerations and angular velocities as measured by sensor unit112as well as the extracted features may be provided or feed into the trained ML model, and the trained ML model may provide an estimation of the current location of vehicle110in real time.

In some embodiments, the trained model may be used together with other sensors such as a camera, a GNSS receiver, a Lidar sensor, radio frequency (RF) sensor, etc., to enhance the accuracy of the location provided by the trained ML model using sensor fusion frameworks. Sensor fusion may be used in the field of accurate navigation and mapping solutions to combine or integrate data acquired by many types of sensors to provide an estimation of the location of the vehicle that is more accurate than each sensor taken alone. In sensor fusion schemes, the sensor's data may be obtained in different sampling rates and may contain various types of information, such as vision, inertial information, position, etc. According to one embodiment, the location data provided by the trained model may be used as a sensory input alongside other data from the sensor. By that, the navigation and mapping solution accuracy may be improved.

According to some embodiments, measurements taken from a plurality of vehicles110that pass in defined area or route120may be stored, for example, in database150, as indicated in operation370. In operation380, anomaly detection techniques may be used to find changes in defined area or route120in a real-time and the map of defined area or route120may be updated. The anomaly detection techniques may include ML algorithms, including unsupervised ML classifiers that may classify new data as similar or different from the training dataset, e.g., K-means, Expectation-maximization, one class support vector machine (SVM), and others. In operation390, a notification regarding a detected change in defined area or route120may be provided to drivers moving in defined area or route120or directly to vehicle110. This technique may allow notifying vehicles110of risks in defined area or route120, as may also be used for tuning the unsupervised ML model (e.g., the unsupervised ML model used for anomaly detection) to achieve higher accuracy by using the acquired data to train or retrain the unsupervised ML model.

Reference is now made toFIG. 5, which presents a DL NN model500during training phase, according to some embodiments of the invention. The model presented inFIG. 5is an example only. Other models may be used. DL NN model500may include at least three layers of nodes, e.g., a convolutional layer510and fully connected layers520and530. In some embodiments, convolutional layer510may implement sigmoid function, fully connected layer520may implement ReLU function, and fully connected layer530may implement a SoftMax function. A loss function540may obtain the predictions550of DL NN model500, and the location or position label560may adjust or tune parameters of DL NN model500through a backpropagation training scheme.

FIG. 6presents the trained DL NN model600, including trained convolutional layer610and fully connected layers620and630. The model presented inFIG. 6is an example only. Other models may be used. Trained DL NN model600may obtain runtime sensor readings of vehicle110and may provide a prediction of the current location or position of vehicle110.

Reference is made toFIG. 7, showing a high-level block diagram of an exemplary computing device according to some embodiments of the present invention. Computing device700may include a processor705that may be, for example, a central processing unit processor (CPU) or any other suitable multi-purpose or specific processors or controllers, a chip or any suitable computing or computational device, an operating system715, a memory120, executable code725, a storage system730, input devices735and output devices740. Processor705(or one or more controllers or processors, possibly across multiple units or devices) may be configured to carry out methods described herein, and/or to execute or act as the various modules, units, etc. or example when executing code725. More than one computing device700may be included in, and one or more computing devices700may be, or act as the components of, a system according to embodiments of the invention. Various components, computers, and modules ofFIG. 1may be or include devices such as computing device700, and one or more devices such as computing device700may carry out functions such as those described inFIG. 3. For example, navigation server may be implemented on or executed by a computing device700.

Operating system715may be or may include any code segment (e.g., one similar to executable code725) designed and/or configured to perform tasks involving coordination, scheduling, arbitration, controlling or otherwise managing operation of computing device700, for example, scheduling execution of software programs or enabling software programs or other modules or units to communicate.

Executable code725may be any executable code, e.g., an application, a program, a process, task or script. Executable code725may be executed by processor705possibly under control of operating system715. For example, executable code725may configure processor705to perform clustering of interactions, to handle or record interactions or calls, and perform other methods as described herein. Although, for the sake of clarity, a single item of executable code725is shown inFIG. 7, a system according to some embodiments of the invention may include a plurality of executable code segments similar to executable code725that may be loaded into memory720and cause processor705to carry out methods described herein.

Storage system730may be or may include, for example, a hard disk drive, a CD-Recordable (CD-R) drive, a Blu-ray disk (BD), a universal serial bus (USB) device or other suitable removable and/or fixed storage unit. Data such as the training dataset of accelerations, angular velocities, and known locations over time, the extracted features, ML model parameters (e.g., weights) and equations, runtime datasets of measured accelerations, angular velocities and extracted features as well as other data required for performing embodiments of the invention, may be stored in storage system730and may be loaded from storage system730into memory720where it may be processed by processor705. Some of the components shown inFIG. 7may be omitted. For example, memory720may be a non-volatile memory having the storage capacity of storage system730. Accordingly, although shown as a separate component, storage system730may be embedded or included in memory720.

Input devices735may be or may include a mouse, a keyboard, a microphone, a touch screen or pad or any suitable input device. Any suitable number of input devices may be operatively connected to computing device700as shown by block735. Output devices740may include one or more displays or monitors, speakers and/or any other suitable output devices. Any suitable number of output devices may be operatively connected to computing device700as shown by block740. Any applicable input/output (I/O) devices may be connected to computing device700as shown by blocks735and740. For example, a wired or wireless network interface card (NIC), a printer, a universal serial bus (USB) device or external hard drive may be included in input devices735and/or output devices740.

In some embodiments, device700may include or may be, for example, a personal computer, a desktop computer, a laptop computer, a workstation, a server computer, a network device, a smartphone or any other suitable computing device. A system as described herein may include one or more devices such as computing device700.

When discussed herein, “a” computer processor performing functions may mean one computer processor performing the functions or multiple computer processors or modules performing the functions; for example, a process as described herein may be performed by one or more processors, possibly in different locations.

In the description and claims of the present application, each of the verbs, “comprise”, “include” and “have”, and conjugates thereof, are used to indicate that the object or objects of the verb are not necessarily a complete listing of components, elements or parts of the subject or subjects of the verb. Unless otherwise stated, adjectives such as “substantially” and “about” modifying a condition or relationship characteristic of a feature or features of an embodiment of the disclosure, are understood to mean that the condition or characteristic is defined to within tolerances that are acceptable for operation of an embodiment as described. In addition, the word “or” is considered to be the inclusive “or” rather than the exclusive or, and indicates at least one of, or any combination of items it conjoins.

Descriptions of embodiments of the invention in the present application are provided by way of example and are not intended to limit the scope of the invention. The described embodiments comprise different features, not all of which are required in all embodiments. Embodiments comprising different combinations of features noted in the described embodiments, will occur to a person having ordinary skill in the art. Some elements described with respect to one embodiment may be combined with features or elements described with respect to other embodiments. The scope of the invention is limited only by the claims.