SYSTEM AND METHOD FOR DETERMINING LOCATION AND ORIENTATION OF AN OBJECT IN A SPACE

A system and method for determining location and/or orientation of a sensor may include, stored in a database a representation of an element in a first space. A mapping between the representation and input from a first sensor may be created. Using the mapping and input from a second sensor in a second space, one or more elements in the database may be identified. A location and/or orientation of the second sensor in the second space may be determined based on the one or more elements.

FIELD OF THE INVENTION

The present invention relates generally to determining location and orientation of an object in a space. More specifically, the present invention relates to determining location and orientation using data from a plurality of modalities.

BACKGROUND OF THE INVENTION

Systems and methods for determining a position or location of an object are known in the art, e.g., Global Positioning System (GPS) and Global Navigation Satellite Systems (GNSS). However, known systems and methods suffer from a number of drawbacks.

For example, the accuracy of known systems is limited to a few meters at best. Additionally, to function properly, known systems require continuous connection between at least two systems, e.g., between a smartphone and a number of satellites. Moreover, known systems and methods do not enable determining a location based on input from a sensor or modality other than a predefined modality. For example, a GPS system cannot determine a location based on images of objects in a space.

SUMMARY OF THE INVENTION

In some embodiments, a representation of an element in a first space may be created and stored in a database. A mapping between the representation and input from a first perception sensor may be created. Using the mapping, and input from a second sensor in a second space, one or more elements in the database may be identified, located or extracted and, using attributes of the one or more elements a location and/or orientation of the second perception sensor in the second space may be determined.

A first mapping may be according to a first condition and a second mapping may be according to a second condition. A mapping may be selected based on a condition related to a sensor. A representation of an element in the database may be based on a first point of view and input from a sensor may be from, or related to, a second, different point of view.

A plurality of mappings for a respective plurality of sensor types may be created. A mapping may be selected based on a type of a sensor. A subset of a set of elements in a database may be selected based on a type of a sensor. A representation of an element may be updated in a database based on input from a sensor. An element represented in a database may be sensed by at least one of: light signals, audio signals, heat, moisture level and electromagnetic waves.

An element in a database may be created based on at least one of: an image, a construction plan, a road map, a blue print, a satellite view, a street view, a top view, a side view an architectural plan and vector data. Information included or stored in the database may be related to, and the location and orientation of a sensor in a space may be determined according to, at least one of: a local coordinate system, a common coordinate system and a global coordinate system.

Information in a database for matching with input from a sensor may be selected based on received or calculated location information. An estimation of a location of a sensor in a space may be determined based on location information related to the sensor. An element may be created based on raw data received from a sensor.

An embodiment may include encoding data from a plurality of sources to produce a unified format representation of the data and including the unified format representation in the database. An embodiment may include utilizing machine learning to identify a set of elements in the database and matching the input with the set of elements. An embodiment may include determining location and orientation of a three dimensional (3D) space by: obtaining a 3D representation of the space using input from a mobile sensor; and determining a location and orientation of the space by correlating the 3D representation with a top view representation of the space.

An embodiment may include projecting a 3D representation on a top view representation. An embodiment may include receiving a set of images and using a relative location in at least some of the images to generate a 3D representation. A top view on which a 3D representation is projected may include at least one of: an aerial image, a satellite image, a structural map, a road map and a digital elevation model (DEM). A portion of a top view representation may be selected based on location information of a space.

A relative location of elements in a space may be determined based on at least one of: input from a sensor and processing a set of inputs having a respective set of points of view. Other aspects and/or advantages of the present invention are described herein.

DETAILED DESCRIPTION

Unless explicitly stated, the method embodiments described herein are not constrained to a particular order in time or to a chronological sequence. Additionally, some of the described method elements can occur, or be performed, simultaneously, at the same point in time, or concurrently. Some of the described method elements may be skipped, or they may be repeated, during a sequence of operations of a method.

Reference is made toFIG. 1, showing a non-limiting, block diagram of a computing device or system100that may be used to determine location and/or orientation of a sensor (or a device connected to the sensor) according to some embodiments of the present invention. Computing device100may include a processor or controller105that may be a hardware controller. For example, computer hardware processor or hardware controller105may be, or may include, a central processing unit processor (CPU), a chip or any suitable computing or computational device. Computing system100may include a memory120, executable code125, a storage system130and input/output (I/O) components135. Controller or processor105(or one or more controllers or processors, possibly across multiple units or devices) may be configured (e.g., by executing software or code) to carry out methods described herein, and/or to execute or act as the various modules, units, etc., for example by executing software or by using dedicated circuitry. More than one computing devices100may be included in, and one or more computing devices100may be, or act as the components of, a system according to some embodiments of the invention.

Memory120may be a hardware memory. For example, memory120may be, or may include machine-readable media for storing software e.g., a Random-Access Memory (RAM), a read only memory (ROM), a memory chip, a Flash memory, a volatile and/or non-volatile memory or other suitable memory units or storage units. Memory120may be or may include a plurality of, possibly different memory units. Memory120may be a computer or processor non-transitory readable medium, or a computer non-transitory memory device or storage medium, e.g., a RAM. Some embodiments may include a non-transitory storage medium having stored thereon instructions which when executed cause the processor to carry out methods disclosed herein.

Executable code125may be an application, a program, a process, task or script. A program, application or software as referred to herein may be any type of instructions, e.g., firmware, middleware, microcode, hardware description language etc. that, when executed by one or more hardware processors or controllers105, cause a processing system or device (e.g., system100) to perform the various functions described herein.

Executable code125may be executed by controller or processor105possibly under control of an operating system. For example, executable code125may be an application that matches features or elements sensed by a sensor to element or features in a database as further described herein. Executable code125may be an application that identifies or extracts features or elements from a databases and associates the features or elements with signatures, identifiers or descriptions thus enabling a mapping between inputs from a variety of sensors and the features in the database.

Although, for the sake of clarity, a single item of executable code125is shown inFIG. 1, a system according to some embodiments of the invention may include a plurality of executable code segments similar to executable code125that may be loaded into memory120and cause controller or processor105to carry out methods described herein. For example, units or modules described herein, e.g., in a vehicle, drone or smartphone and/or in a server, may be, or may include, controller or processor105, memory120and executable code125.

Storage system130may be or may include, for example, a hard disk drive, a universal serial bus (USB) device or other suitable removable and/or fixed storage unit. As shown, storage system130may include a database131that may include element representations132(collectively referred to hereinafter as element representations132or individually as element representation132merely for simplicity purposes).

Objects in storage system130, e.g., element representations132and other objects in database131may be any suitable digital data structure or construct or computer data objects that enables storing, retrieving and modifying values. For example, element representation objects132may be non-volatile memory segments, files, tables, entries or lists in database131, and may include a number of fields that can be set or cleared, a plurality of parameters for which values can be set, a plurality of entries that may be modified and so on. For example, a location, color, shape, unique identification or signature of an object, element or feature represented by an element representation132may be set, cleared or modified in the element representation132that may be associated with the actual element in a space, region or location.

Attributes of an element (object or feature), e.g., included or described in an element representation132, may be, or may include any relevant information. For example, attributes of an element may be, or may include, a location of the element, an orientation of the element in space, a color of the element, a temperature of the element, a size of the element, a shape of the element or any other characteristics of the element.

The phrase “element in a space” and term “element” as used herein may relate to any object, element or feature that can be sensed or identified by any sensor or modality. For example, an element (in a space) may be a house, a pole or a tree, e.g., sensed by a radar or a LIDAR system. In another example, an element can be a color of a wall, e.g., sensed by a camera. An element or feature can be, for example, a change of color, for example, if a first part of a wall is green and a second part of the wall is blue then the line separating the green part from the blue part may be a feature, e.g., sensed by a camera, as described. A color change feature may be recorded and used for determining a location as described. In yet another example, a temperature at a location can be a feature (e.g., sensed using an infrared (IR) sensor). Any object or phenomena in space may be an element as referred to herein. For the sake of simplicity, the term element will mainly be used herein, it will be understood that an element as referred to herein may mean, or relate to, any object or phenomena that can be sensed by any sensor.

Additionally, or alternatively, the term “element” may refer herein to features and elements that may not necessarily be directly perceived by a human observer and/or by any type of sensor. For example, as elaborated herein (e.g., in relation toFIG. 2), embodiments of the invention may include one or more AI engines (e.g., localization AI engine ofFIG. 2), adapted to extract one or more elements (referred to herein as “location indicative elements”) that may that may be characteristic, or defining of a geographical location, but nevertheless may not be associated with any specific physical object and/or may not be perceivable by a human observer and or by any type of sensor.

A unique identifier, reference or signature associated with an element, feature (or element representation132) may be unique within a specific instantiation of the invention, but not be unique when compared with the universe of numbers of data stored on all existing computer systems. For example, a unique identifier, reference or signature associated with an element may be computed based on attributes of the element, including a location of the element as well as other attributes, thus, for each real or actual element in space, a unique identifier may be calculated.

Content may be loaded from storage system130into memory120where it may be processed by controller or processor105. For example, element representations132may be loaded into memory120and used for matching them with input from a sensor as further described herein.

In some embodiments, some of the components shown inFIG. 1may be omitted. For example, when computing device is installed in a vehicle, drone or smartphone, memory120may be a non-volatile memory having sufficient storage capacity and thus storage system130may not be required.

I/O components135may be used for connecting (e.g., via included ports) or they may include: a mouse; a keyboard; a touch screen or pad or any suitable input device. I/O components135may be or may be used for connecting, to computing device100, devices or sensors such as a camera, a LIDAR, a radar, a heat sensing device, a microphone, an infra-red sensor or any other device or sensor. I/O components may include one or more screens, touchscreens, displays or monitors, speakers and/or any other suitable output devices. Any applicable I/O components may be connected to computing device100as shown by I/O components135, for example, a wired or wireless network interface card (MC), a universal serial bus (USB) device or an external hard drive may be included in I/O components135.

A system according to some embodiments of the invention may include components such as, but not limited to, a plurality of central processing units (CPU) or any other suitable multi-purpose or specific processors, controllers, microprocessors, microcontrollers, field programmable gate arrays (FPGAs), programmable logic devices (PLDs) or application-specific integrated circuits (ASIC). A system according to some embodiments of the invention may include a plurality of input units, a plurality of output units, a plurality of memory units, and a plurality of storage units. A system may additionally include other suitable hardware components and/or software components. In some embodiments, a system may include a server, for example, a web or other server computer, a network device, or any other suitable computing device.

Some embodiments of the invention may automatically and/or autonomously, create a representation, view and/or mapping of a space based on input from a set of different sensing devices. A representation, view and/or mapping of a space (and elements, features or objects therein) may be created, e.g., in database131, by automatically creating element representations based on input from sensors of any type. A mapping between element representations and input from sensors may be automatically created. Accordingly, a system and method may automatically, based on input from a plurality of different sensors or modalities, create a representation, view and/or mapping of a space, e.g., such that objects or features in the space are usable for determining a location and/or orientation of a sensor. For example, after a mapping and representations are created, input from a sensor may be mapped to representations and the representations may be used to determine a location and/or orientation of the sensor.

Some embodiments of the invention may provide a positioning solution based on matching features or objects captured by perception sensors to features or objects represented in a database. Some embodiments of the invention may match an element as seen or captured, from/by a first point of view or first modality with an element or feature as seen/captured from/by a second point of view or modality. Accordingly, unlike known location or positioning systems and methods, embodiments of the invention are not limited to specific sensors, modalities or technologies.

For example, representations of elements, objects or features may be included in a database based on a top-view (e.g., satellite images) and these (representations of) elements, objects or features may be correlated or matched with elements, objects or features captured or seen using street-view images, e.g., captured by a camera in a vehicle. For example, elements in a top-view may be matched with elements in a ground surface view.

Some embodiments of the invention may enable using different sensor technologies without the need to collect new data. Additionally, some embodiments of the invention may be adapted to determine a location and orientation in various, different scenarios and/or conditions, e.g., different road condition, weather and visibility conditions.

As described, some embodiments of the invention may use auto encoding features to learn a correspondence or correlation between multiple instances of a scene, correspondence or correlation between multiple points of view and/or correspondence or correlation between representations of a scene by a plurality of modalities. The resulting system thereby learns localization features which are both discriminative or distinctive and invariant.

Reference is made toFIG. 2, an overview of a system200and flows according to some embodiments of the present invention. As shown, a system200may generally include at least one server210and at least one mobile device230. Server210may be, or may include one or more computing devices and/or modules that, for the sake of simplicity, will be referred to herein as server210. As shown, server210may include a dataset215, an analytics toolbox216and a database (DB)217. As further shown, server210may include, or be operatively connected to, a localization artificial intelligence (AI) engine or unit220.

Mobile device230may be a vehicle, a mobile computing device such as a smartphone or laptop, a drone or any other suitable computing device. As shown, mobile device230may include a DB232and an onboard localization and orientation (OLO) unit233.

System200or components of system200may include components such as those shown inFIG. 1. For example, server210, analytics toolbox216and AI unit220may be, or may include components of, computing device100. Mobile device230and OLO unit233may be, or may include components of computing device100. Although for the sake of clarity, a single mobile device230is shown inFIG. 2, it will be understood that a system200may include a large number of mobile devices230. For example, server210may communicate with any (possibly very large) number of vehicles that may include the elements in mobile device230.

According to some embodiments, system200may be scaled up by deploying a plurality of servers210that may be communicatively connected among themselves. Additionally, system200may be scaled up by including a large numbers of mobile devices230.

As described herein, system200may be adapted to calculate, determine and/or provide (e.g., for a mobile device230), an exact location and orientation (e.g., of mobile device230) in space.

As shown inFIG. 2, input from one or more sensors included in, or connected to mobile device230may be received, as shown by sensor data231. According to some embodiments, sensor data231may be matched, in real-time or near-real time, with information produced by server210and provided to mobile device230. For example, information produced by server210and provided to mobile device230may be stored in DB232. Generally, any data or information included in database131or217may be communicated to, and stored in, DB232.

For example and as described herein, based on an estimation of a location of a vehicle, some of element representations132may be downloaded to DB232of a specific mobile device230, such that DB232only stores element representations132that describe elements that are physically located near the specific mobile device230.

The terms “real-time” (also referred to in the art as “real time”) and “near-real time” as referred to herein may relate to processing or handling of events at the rate or pace that the events occur or received (possibly defined by human perception). For example, a system according to embodiments of the invention may match input from sensor311(that may be a mobile sensor) with at least one element representation132, in real-time, e.g., within milliseconds or other very brief periods so that location and/or orientation of sensor311are made available or achieved virtually immediately.

Dataset215may include information from any system or source. For example, dataset215may include information extracted from images, construction plans, road maps, blue prints, architectural plans and vector data of elements in a region or space. Dataset215may include information extracted from satellite images or view, street view, top view and a side view of elements in a region or space.

AI unit220may use any method or system that includes artificial intelligence (AI) or machine learning (e.g., deep learning). Generally, machine learning (ML) or deep learning may relate to computerized systems or methods that perform a task without receiving or requiring specific instructions. For example, AI unit220may be a system that may be adapted to create at least one model221based on sample or training data. Analytics toolbox216may use the model221to make predictions or decisions without being explicitly programmed to perform the task of mapping input (e.g., from sensors310and311) to element representations132as described.

A deep learning or AI model may use a neural network (NN). A NN may refer to an information processing paradigm that may include nodes, referred to as neurons, organized into layers, with links between the neurons. The links may transfer signals between neurons and may be associated with weights. A NN may be configured or trained for a specific task, e.g., pattern recognition or classification. Training a NN for the specific task may involve adjusting these weights based on examples. Typically, the neurons and links within a NN are represented by mathematical constructs, such as activation functions and matrices of data elements and weights. A processor, e.g. CPUs or graphics processing units (GPUs), or a dedicated hardware device may perform the relevant calculations.

According to some embodiments, localization AI engine220may be adapted to create and/or train one or more ML models221, such as deep-learning ML models, based on respective one or more sources of data. For example, localization AI engine220may receive (e.g., from dataset215) data that may include, for example, images, construction plans, road maps, blue prints, architectural plans, aerial images or any vector data. Additionally or alternatively, localization AI engine220may receive data originating from one or more sensors (e.g., a camera, a radar, a LIDAR sensor, an infrared sensor, and the like) such as element310ofFIG. 3. Localization AI engine220may be adapted to create or train a first ML model221corresponding to a first type of data (e.g., included in dataset215) and produce a second ML model221corresponding to a second type of data (e.g., data originating from the one or more sensors).

According to some embodiments, localization AI engine220220may be adapted to train at least one deep learning model221, based for example on data in dataset215. Model221may be adapted to identify or predict, as commonly referred to in the art, one or more features or elements that may be included in dataset215.

For example, dataset215may include a plurality of data elements such as satellite and/or aerial photographs. AI engine220may produce a deep learning model221that may be or may include an autoencoder neural network. As known in the art, autoencoder neural networks may be trained so as to produce a compressed representation or an encoding of elements included in an input dataset. Thus, in a training stage, AI engine220may use a subset of the aerial photograph data elements as a training subset, to train model221so as to produce a representation132(e.g., compressed representation) or encoding of one or more elements (e.g., tree, houses, etc.) included in the training subset. In a subsequent inference stage, AI engine220may produce a compressed representation132or encoding of one or more elements (e.g., houses, poles, trees, etc.), that may be included in data elements (e.g., element218) of dataset215beyond the training subset. Additionally, or alternatively, in the inference stage, AI engine220may identify one or more elements or features (e.g., houses, poles, etc.) that may be included in data elements (e.g., element218) of dataset215beyond the training subset.

In another example, dataset215may include a plurality of data elements that may originate from sensors (e.g., element310ofFIG. 3) that may be located at ground level, including for example, cameras, LIDAR sensors, radar sensors, etc. AI engine220may use a subset of the plurality of sensor (e.g.,310) data elements as a training subset, to train model221so as to produce a representation132(e.g., a compressed representation) or encoding of one or more elements or features (e.g., trees, cars, houses, etc.) that may be included in data elements (e.g., element231, element218, data of sensor310) beyond the training subset.

According to some embodiments of the invention, one or more (e.g., each) data element218of dataset215may be associated with a geographical position and/or orientation. For example, a data element218may be an aerial photograph, and may include, or may be associated with a location or position and/or an orientation at which the photograph was taken. In another example, a data element218may be an architectural plan (e.g., of a house, a constructed monument, and the like), and may be associated with a geographical location of a respective constructed element. In yet another example, a data element218of dataset215may originate from a sensor (e.g., an image of radar) and may include or be associated with a location at which the respective sensor data was obtained.

According to some embodiments, localization AI engine220may be configured to produce at least one ML model221, adapted to predict one or more elements320or features that may be included in raw data218and may best correspond to geographical location. It may be appreciated by a person skilled in the art that such elements may not necessarily correspond to actual, physical elements (e.g., a house, a pole) that may, for example, be perceived by a human eye. In some cases, such elements may be regarded as incoherent or amorphous by a human observer. Such elements are herein referred to as “location indicative” elements (e.g., element325ofFIG. 3). For example, embodiments of localization AI engine220may extract at least one location indicative element325, that may be or may include a combination (e.g., a numerical combination, such as a weighted sum) of a first number (e.g., representing the color of the sky) and a second number (e.g., representing a pattern of vegetation). Localization AI engine220may use the extracted at least one location indicative element325as a characteristic or an indicator of a geographical location (e.g., indicate whether an image has been taken in a mountainous environment or in a desert).

Using the at least one model221, analytics toolbox216may create representations, descriptions and/or identifications of features and/or elements in a region or space. In other words, analytics toolbox216may perform inference of the at least one model221on a plurality of data elements.

Pertaining to the example of aerial images, analytics toolbox216may perform inference of at least one model221on one or more aerial images originating, for example from dataset215(e.g., data element218), and may map or associate at least one representation, description and/or identification (e.g., a label of an identified object, such as a house) with a respective elements included in the aerial images.

In another example, analytics toolbox216may receive a model221, adapted to analyze a specific type of input data, such as satellite images. In such embodiments, model221may have been trained on a first set of satellite images taken over one or more first locations, and adapted to identify objects (e.g., road signs, buildings) in the images. Analytics toolbox216may infer model221on one or more satellite images (e.g., data elements218) of dataset215, taken over one or more second locations, to identify objects in the one or more second locations. As elaborated herein, data elements218may be associated with specific geographical locations. Therefore, analytics toolbox216may produce one or more element representations132corresponding to the one or more identified objects (e.g., road signs, buildings, etc.), and the one or more element representations132may be associated with the specific geographical locations. Said element representations132associated with specific geographical locations may herein be referred to as road codes (e.g., element250).

In yet another example, localization AI engine220may produce an ML model221that may be adapted to receive at least one input data element of a specific type or modality, such as a specific type of sensor (e.g., element310ofFIG. 3). ML model221may be adapted to extract or predict from the at least one received data element one or more location indicative elements320, as elaborated herein. Analytics toolbox216may be configured to receive at least one data element, such as data element218from dataset215, that may correspond to the same specific type or modality. Analytics toolbox216may use ML model221to extract at least one location indicative element325that may be included in input data element218, and may be a characteristic of a geographical location. As elaborated herein, data element218may be associated with a geographical location or position. Therefore, analytics toolbox216may create an element representation132that may also be associated with the same geographical position. In other words, analysis toolbox216may produce, from the incoming data element218at least one road code250that may be associated with an location indicative element325, based on ML module221.

The term “mapping” may refer herein to an association of a label or a representation of an element132with one or more data elements (e.g., one or more images and/or one or more portions thereof) that may be, for example, included in dataset215and/or obtained from a sensor (e.g., element310ofFIG. 3).

According to some embodiments, analytics toolbox216may update DB217so as to include said mapping or association between element representations and corresponding input data (e.g.,218and/or data of sensor310).

In some embodiments, one or more representations, descriptions and/or identifications of features and/or elements may be provided to mobile device230. For example, and as shown inFIG. 2, mobile device230may receive one or more road codes250, and may be stored in DB232.

According to some embodiments, OLO unit233may extract road-codes250and/or other representations, descriptions and/or identifications of features and/or elements from DB232, match or correlate them with sensor data231(produced by sensors in mobile device230) and, based on the matching, determine its location and/or orientation260in relation to the relevant representations. Location and orientation260may be presented to a user, e.g., using a screen, speakers and the like.

As shown by the arrow connecting OLO unit233and analytics toolbox216, OLO unit233may update analytics toolbox216, e.g., to remove an element that is no longer present in a region, add an element that is missing in DB217, modify a description of an object and the like.

Reference is additionally made toFIG. 3which shows components and flow according to some embodiments of the invention. As shown, an element representation132and input from a first sensor310may be used to create a mapping330between the input and the representation. As further shown, using the mapping330, input from a second sensor311may be matched with the element description132. Sensors310and311may for example be mobile sensors, e.g., installed in a car, aircraft, ship or bike or hand carried by a user, e.g., included in a smartphone.

Additionally, or alternatively, at least one sensor310may be adapted to obtain at least one input data element that may pertain to at least one location indicative element325. As elaborated herein, location indicative element325may not pertain to any specific physical object (such as house320or pole323), but may be a produce of ML model221. According to some embodiments, location indicative element325may be unintelligible or incoherent to a human observer, but may be recognized by localization AI engine220as indicative of geographical locations. In such embodiments, analytics toolbox216may perform mapping between a representation132of location indicative element325, and the input data of the at least one sensor310.

Mapping330may be, or may include any information or logic that, provided with input from a sensor, produces at least one value, indication, pointer or reference that enables identifying, locating or extracting, at least one element representation132from a database. Otherwise described, mapping330may be, or may be viewed as, a conversion unit, data or logic that converts input from a sensor to an element representation132object.

For example, a first sensor (e.g., element311, such as a radar) may be adapted to sense an object (e.g., element323, such as a pole). First sensor311may be adapted to subsequently output a signal or a data element corresponding to the sensed object. In the example of the pole, first sensor311(e.g., the radar) may produce one or more or more signals or data elements that correspond to one or more (e.g., a set of) values or readings that represent a frequency, an amplitude or other attributes of RF radiation or electromagnetic waves, reflected from pole323. The set of values may be mapped to, or associated with, an element representation object132, accordingly, when the set of values is received at a later stage (e.g., by another radar sensor311), the mapping or association can be used to map the values to the element representation132object of pole323.

As described, mapping330may generate a signature based on a set of values received from a sensor as described and the signature may be used for finding the correct element representation132object, the signature may be used as a key for finding an element representation132object in a database, e.g., in DB232. For example, OLO233may examine sensor data231(input from a sensor), calculate or produce a signature based on the input and use the signature to find an element representation132object in DB232. Once an element representation132object is found, attributes such as the location of the element may be extracted from the element representation132object.

For example, an element in space340may be a corner of a house320, a house321, a pole or tree323or a change of color322, e.g., element or feature322may be a line separating a white part of a wall from a black part. Elements, objects or features320,321,322,323and325may be represented in database131using element representations132. Element representations132may include representations or encoding of elements that may originate from AI model221. For example, AI model221may be or may include an autoencoder model, having a bottleneck hidden layer, which may be adapted to produce a representation (e.g., a compressed representation) or encoding of elements included in an input data set.132.

Generally, an element representation132may include any metadata related to an element, object or feature. The terms element, object and features as referred to herein may mean the same thing and may be used interchangeably herein. For example, an element representation132may include a location (e.g., using coordinate values as known in the art), a color, a size, a shape and the like. For example, an element representation132of house321may include the house's location, an element representation132of pole323may include the pole's height and an element representation132of color change322may include the location and length of a line that separates two colors on a wall as well as an indication of the two colors.

Generally, a coordinate system enables using values to represent a location and/or orientation in a space. For example, coordinates in a Cartesian coordinate system are tuples that represent a location of a point in space. A coordinate system may be standard or global, e.g., known and used by various systems and methods or it can be a relative one. For example, a relative coordinate system may represent a location of an element in space with respect to sensor311, e.g., a relative coordinate system may be used to indicate the location of house corner320with respect to sensor311, e.g., the origin (e.g., the point 0,0,0) of a relative coordinate system may be sensor311such that the relative locations of pole323and house321are represented with respect to the location of sensor311.

It is noted that the point of view of sensor310may be different from the point of view of sensor311, for example, sensor310may be installed on a rod extending from a roof of a car traveling from north to south near house321and sensor311may be placed on a bumper of a car traveling in the opposite direction near house321, as described, an element representation132of an element in a database may be based on a point of view (or modality or sensor) that is different from the point of view, sensor type or modality of sensors310and311, accordingly, embodiments of the invention enable matching representations of elements that are derived using different modalities, input types or sensors types. Accordingly, sensors (e.g., sensors310and311) may be installed in different vehicles or in different stationary objects.

In some embodiments, a method of determining location and orientation of a sensor may include storing or including, in a database, a representation of an element in a first space and creating a mapping or correlation between the representation and input from a first perception sensor (e.g., sensor310). For example, based on a satellite image, a representation132of house321seen in the satellite image, may be included as an element representation132in database217(e.g., element131ofFIG. 1), thus forming a representation of an element in a first space. Subsequently, input from a camera (first perception sensor, e.g., sensor310), e.g., in a vehicle traveling by house321, may be received (forming the representation and input from a first perception sensor). An embodiment may examine the image, identify house321therein and define a mapping (e.g., mapping330) between the image representation of house321in the image and the element representation132in database217(e.g.,131) which describes house321. It will be noted that a sensor that is a conventional (visible light) camera and that produces an image is just one of the types of applicable sensors. For example, sensor310may be a LIDAR device, a radar, a heat sensing device, a microphone, an infra-red (IR) sensor and the like.

Processing and operations described herein may be performed by a server (e.g., server210) and/or by a unit in a vehicle (e.g., OLO233) or a server and a unit in a vehicle may collaborate in performing methods described herein. For example, server210may include in a database a representation of an element in a first space, e.g., in the form of an element representation object132and the server may further create a mapping330between the representation and input from a sensor (e.g.,310) and provide the mapping to OLO233which may use the mapping and input from a second perception sensor (e.g.,311) to identify one or more elements in a database (e.g., in DB232) and use attributes of elements identified in the database to determine a location and/or orientation of mobile device (e.g., vehicle)230.

It is noted that, e.g., in the above example involving a camera, once a mapping330is created as described, any image of house321captured by practically any camera or image acquisition device may be mapped to the element representation132of house321in database131(e.g., element217and/or element232). For example, one or more attributes of house321such as size, shape, height, color, contours and the like may be taken into account when creating or defining mapping330, such that any image of house321may be mapped, using mapping330, to the correct element representation132in database131(e.g., element217and/or element232), that is, to the specific element representation132that describes house321.

It may be appreciated that although the example of a house (e.g., elements320,321) is used ubiquitously herein, similar logic may be applied to any other physical feature or element (e.g., color change322, pole323), or on non-physical elements, such as location indicative element325.

In some embodiments, a method of determining location and orientation of a sensor may include using a mapping and input from a second perception sensor, in a second space, to locate or identify one or more elements in the database. For example, using mapping330, input from sensor311(second sensor) may be used to locate or identify house321in DB232. For example, an image of house321, as captured by sensor311(that may be a camera or other sensing device) may be processed using mapping330to produce a result that may be used to locate element representation132that describes house321, in database131or in DB232.

In some embodiments, a method of determining location and/or orientation of a sensor may include using attributes of identified (e.g., in a database) one or more elements to determine a location and orientation of the second perception sensor in the second space. For example, provided with an image of house321captured by sensor311(the second sensor) and using mapping330, OLO unit233may produce a reference, key or signature that can be used for extracting, from DB232, the element representation132object that describes house321.

As described, the element representation132object that describes house321may include any metadata or attribute, e.g., an attribute or metadata in the element representation132object may be the exact location of house321in the form of coordinates values in a global or known coordinate system. Accordingly, exact locations of elements320,321,322,323and325may be obtained by OLO unit233thus, e.g., using triangulation, OLO unit233may accurately determine its location and/or orientation.

In some embodiments, a method of determining location and orientation of a sensor may include creating a first mapping according to a first condition and creating a second mapping according to a second condition and selecting to use one of the first and second mappings according to a condition.

For example, sensor310may be used to sense house321in daylight and then at night and two mappings330may be created, one to be used during the day and another for night time. Similarly, mappings330may be created for summer, winter, storm, specific dates, time of day and so on. Similarly, a first mapping330of input from sensor311when sensing pole323on a rainy day to an element representation132object may be created and a second mapping, to the same element representation132may be created for input from sensor311when sensing pole323on a sunny day. Accordingly, a mapping may be based on a condition, time of day, date and so on. As described, based on a condition, a mapping may be used, e.g., if pole323is sensed or “seen” by sensor311during day time then the day time mapping may be used and, if the pole is sensed by sensor311during the night (thus input from sensor311may be different) then a night mapping may be selected. Accordingly, a mapping may be created and dynamically selected based on a condition. For example, OLO233may select a first mapping if the weather is fine and a second mapping in a storm, e.g., OLO233may select a first mapping in a first condition (e.g., cold weather or rainy day) and select a second mapping in a second condition (e.g., warm weather or sunny day).

In some embodiments, elements (e.g., element representations132) may be classified. For example, element representations132may be classified according to a suitability for sensor type (e.g., a first class of elements may be suitable for use with a camera and a second class of elements may be suitable for use with a LIDAR sensor or device). Element representations132may be classified according to a condition, e.g., elements best detected at night may be associated with a first class and elements best detected in sunlight may be associated with a second class. Similarly, classification of elements may be according to weather conditions, location (e.g., elements in a tunnel may be classified differently from elements in a farm and so on. Any aspect or condition may be used for classifying elements. Accordingly, an embodiment can automatically select a set of elements to be used for determining location and/or orientation based on any condition or aspect. For example, if it rains when location of sensor311is to be determined then a set of elements classified as best for poor visibility may be selected and used as described herein for determining the location and/or orientation of sensor311.

In some embodiments, a representation132of an element in database131may be based on, or related to, a first point of view and input from a sensor for which location and orientation is determined is related to a second, different point of view.

For example, a first point of view may be used for creating mapping330and a second, different point of view may be used for determining location and orientation of a sensor. For example, the point of view of sensor310, when sensing house321to create mapping330as described, may be different from the point of view of sensor311when its location and orientation are determined based on identifying house321as described.

In some embodiments, a plurality of mappings330may be created for a respective plurality of sensor types or modalities. In some embodiments, a method of determining location and orientation of a sensor may include selecting one of a plurality of mappings330based on the type of the sensor.

For example, in a first case, mobile device230may inform server210that it includes a camera (sensor type), in response, server210may search database131or DB217for elements or features that can be sensed by a camera, e.g., colors, road signs and the like. Next, server210may send to mobile device230(e.g., as road codes element250) the mappings for the selected elements, e.g., the mapping330of color change322may be downloaded to mobile device230since it's best suited for sensing by a camera. In a second case, mobile device230informs server210that it includes a LIDAR system, in this case, server210may send mapping330for pole323but avoid sending the mapping for feature322since a change of color on a surface cannot be accurately sensed by a LIDAR.

Accordingly, an embodiment may create a plurality of mappings330for a respective plurality of sensor types and automatically or dynamically select a mapping to be used based on the type of sensor for which location and/or orientation are determined, an embodiment may further dynamically or automatically select a subset of elements in a database, wherein the subset selected based on the type of the second sensor, that is, both mappings and element representations132may be selected such that the set of element representations132and mappings330downloaded to DB232is the optimal set for the sensor used. Of course, if mobile device includes a number of different sensors or different types of sensors (e.g., camera and LIDAR) then element representations132and mappings330for both camera and LIDAR may be automatically selected and downloaded to mobile device230as described.

Accordingly, embodiments of the invention may select, from a large set, a subset of element representations132and mappings330based on a location of a sensor and based on attributes of the sensor, e.g., the sensor type and/or suitability of the sensor to sense different or specific elements in the vicinity of the sensor. As described, the set may also be selected based on a condition.

In some embodiments, an element representation132in one or more of database131, DB217and DB232may be updated based on input from a sensor. For example, starting from a satellite image, a plurality of circles may be identified, and their respective diameters and locations may be included in their respective element representations132. Next, an image may reveal that the circles are the top surfaces of poles, in such case, the element representations132already in database131may be updated to reflect the fact that the elements or poles. Any metadata related to an element or feature may be updated, or modified as described. For example, in the above example of poles, updating the element representations132may include indicating which sensors can sense them, in what conditions the elements can be sensed and so on.

In some embodiments, e.g., if input from sensor311cannot be mapped to any element representation132, a new element representation132may be created in database131. For example, if pole323was recently placed by a road then input from the first sensor310that passes by pole323may cause controller or processor105to add a new element representation132to database131. As described, input from additional sensors passing by pole323may be used to verify that pole323is present and/or map the element representation132of pole323to other sensor types as described.

Element representations132may be removed from database131. For example, after pole323is removed from the road side, sensor311does not sense it when passing where pole323was placed, by comparing input from sensor311to data in DB232, controller or processor105may identify that pole323is no longer sensed and may remove the relevant element representation132from all databases in system200. Accordingly, embodiments of the invention continuously and autonomously update a representation of space such that it truly reflects the current state of the space.

The advantages of automatically and autonomously creating digital representations of elements, objects or features as described may be appreciated by those having ordinary skill in the art. For example, assuming a new sensor, e.g., an IR sensor, is introduced to the field or market, e.g., input from such a type of IR sensor has never been received by system200. When first receiving input from the new IR sensor, system200may automatically create element representations132based on the input. An embodiment may then create a mapping between features, elements or objects sensed by IR sensors and element representations132, e.g., during a training session. An embodiment may associate or map element representations132created based in IR sensors with other elements in a database such that information related to a newly introduced sensor are identified and characterized using input from other sensors and/or input from other sources, e.g., maps and the like. As described, element representations132added may be used to determine a location and/or orientation of IR sensors. Accordingly, a system and method need not be configured or modified in order to adapt to new sensor types, rather, systems and methods of the invention can automatically adapt to newly introduced sensors or modalities.

A versatility with respect to sensor types or modalities provided by embodiments of the invention cannot be provided by known systems and methods. The versatility with respect to sensor types or modalities is an improvement current or known systems and methods which are restricted to specific sensors or modalities. For example, sensor311may be a microphone and the sound of a power generator in a factory (e.g., intensity and frequency) may be used for creating an element representation132object that may be used to determine location and/or orientation as described. Similarly, any phenomena that can be sensed by any sensing device may be represented by an element representation132in database131and used as described, e.g., element representations132objects may be created and/or updated based on optical data, acoustical data and radio frequency data.

By supporting any modality or sensor type as described, embodiments of the invention improve the field of location sensing or determination by providing scalability and configurability that cannot be provided by known systems or methods. For example, a user who just added a LIDAR system to his car can immediately receive location and/or orientation information based on the newly added LIDAR system and based on element representations132that are relevant to LIDAR systems, e.g., based on element representations132and a mapping that were previously created or updated based on input from LIDAR systems. A system and method may be trained to support any sensor type or modality as described. It will be noted that once a system is trained to use a specific sensor or modality, it may be used to determine a location and/or orientation of a sensor even in places or spaces where no input from the specific sensor type was received, e.g., using a mapping as described, any object in any space may be identified as described.

Generally, an element or feature as referred to herein may be any object, element or feature that can be sensed by any sensor or modality, e.g., an element or feature may be sensed by, or based on, a light signal, an audio signal, temperature or heat, moisture level, electromagnetic waves or any other phenomena that can be sensed. For example, moisture near a carwash place or water park may be sensed, heat from a white wall (as oppose to heat from a black wall) may be sensed and so on. For example, a first element representation132may describe an object sensed by sonar, a second element representation132may describe an object sensed based on heat (e.g., using IR) and so on.

Element representations132may be created based on any source, modality or type of data, e.g., they may be created based on an images, construction plans, road maps, blue prints, architectural plans, aerial images or any vector data. For example, element representations132may be created, e.g., by server210, based on a satellite image or view, a street view footage or images, a top view and a side view all of which may be obtained by server210from any source, e.g., the internet, authorities and so on. Accordingly, any modality (and, as described, any data from any point of view) may be used for creating element representations132. For example, an element representation132in database131, related to an object in space may be created based on a mark in a map, may then be updated based on an image of the object, then updated based on input from a LIDAR device and so on. As described, element representations132in database131may be found, identified and/or retrieved based on input from a sensor (of any type) using a mapping as described.

In some embodiments, a global or common coordinate system may be used. For example, as described, an element representation132may include location information in the form of coordinates in a first coordinate system, in case a second, different coordinate system is used by mobile device230, an embodiment may convert or transform location information such that it is presented to a user according to a preferred or selected coordinate system. In some embodiments, an element representation132may include location information (e.g., coordinates) related to one or more coordinate systems, e.g., related to a global coordinate system and a number of different local coordinate systems, in other embodiments, a conversion logic may be used to translate or transform location information as required.

In some embodiments, the information in a database that is matched with input from a sensor as described is selected based on received or calculated location information. In some embodiments, an estimation of a location of the sensor in a space, area or region is determined based on location information related to the sensor. For example, based on GPS data, a cellular tower's location information or Wi-Fi access points identifications, at list the area where sensor311is in may be determined and server210may select to download to DB232elements in the area. Accordingly, only relevant elements in a space or region may need to be stored by mobile device230thus eliminating the need of mobile device230to store, and search through, large amounts of data.

In some embodiments, mapping330may map or correlate any input from a sensor to an element representation132, e.g., raw data received from sensor311when sensing corner320may be processed using mapping330to produce a signature, pointer or reference that can be used to identify the element representation132that describes corner320.

In some embodiments, data received from a plurality of sources, e.g., a plurality of sensor types, may be encoded to produce a unified format representation of the data and the unified format representation may be included in the database, e.g., in one or more element representations132. By encoding different data types coming from different sensor types into a unified format representation, an embodiment can represent a single element or feature in space based on input from a plurality of different modalities. For example, an element representation132may be created based on a map (first source) and a satellite image (first modality) and based on input from a mobile sensor such as a mobile camera (second modality), such creation of an element representation132based on input from a plurality of sources, sensor types and modalities may be done using a unified format representation. In some embodiments, a unit (not shown) connected to a sensor in mobile device230may convert raw data received, e.g., from sensor311, to a unified format representation (that may be included in, or associated with, element representations132, e.g., in the form of metadata as described), thus, efficiency of a search in DB232for an element representation132is improved and speed of operations described herein is increased.

Adapted to use input from any type of sensor, embodiments of the invention improve the technology which is currently confined to a limited, preconfigured set of sensors or modalities, embodiment of the invention provide a practical application of a process for determining a location and/or orientation of a sensor (or vehicle or other entity that includes the sensor), for example, a practical application of such process includes OLO233, DB232and at least one sensor311installed in a car such that a system can provide the driver with his or her location based on input from a variety of sensors that may be of any type or modality.

In some embodiments machine learning may be used to match input from a sensor with elements. For example, an iterative learning process may include examining a set of inputs, from one or more sensors, related to an object and generating a model that, given input of a sensor sensing the object maps the input to an element representation132that is associated with the object.

Reference is made toFIG. 4, a flowchart of a method according to illustrative embodiments of the present invention. As shown by block410, a representation of an element may be included in a database. For example, a plurality of element representations132may be included in database131. As shown by block415, a mapping between the representation and input from a first sensor may be created. For example, based on input from sensor310, when sensing pole323, an embodiment may create a mapping330such that, using the mapping, input from a sensor near pole323is mapped to an element representation132that describes pole323.

As shown by block420, the mapping may be used to map input from a second sensor, in a second space, to identify one or more elements in a database. It will be noted that the first and second sensors may be of the same type, or they may be of different types. The first and second spaces may be the same space in different times or they may be different spaces. For example, if a mapping enables identifying pole323in DB232then similar or identical poles (in another place or space) may be identified using the mapping created for pole323. As shown by block425, using attributes of the identified one or more elements, a location and/or orientation of a sensor may be calculated and/or determined. For example, by identifying a number of elements around sensor311as described, the exact locations of these elements may be known (e.g., since it may be included in their respective element representation132objects) thus the location of sensor311may be determined, either globally or with respect to the identified elements. For example, a location provided to a user by an embodiment may be in the form of coordinate values and/or in the form of an indication of the location on a map, e.g., as done by navigation systems. An orientation provided to a user may be in the form of an image or avatar (e.g., of a car) where the image or avatar is shown, on a map, such that the orientation (e.g., direction of movement) are clearly shown.

Some embodiments may continuously update, add and/or remove element representations132, e.g., based on iteratively receiving input from sensors and updating metadata in, or of, element representations132objects or other information in database131. For example, recorded inputs from a plurality of sensors311that travel near elements, objects or features320,321,322,323, and325over a possibly long time period, may be anchored or fixed in using a common coordinate system such that accuracy, e.g., of the location of element320(as included in metadata of the relevant element representation132) verified and/or is increased. For example, using the locations of sensors311when they sense feature322, a set of inputs from these sensors may be correlated or brought into a common representation, e.g., a common coordinate system.

Accordingly, the set of inputs may be used for increasing accuracy of location and/or orientation of elements, thus, accuracy of location and/or orientation of a sensor provided by embodiments of the invention is increased.

In some embodiments, location and/or orientation of a 3D space may be determined by obtaining a 3D digital representation of a space using input from a mobile sensor; and determining a location and/or orientation of the space by correlating the 3D representation with a top view representation of the space. For example, sensor311may be a video or other camera used for capturing a set of images of a street (e.g., street view footage). The set of images may be used to create a 3D representation of a space that includes or contains the street. The 3D representation may then be correlated with a top view of the street, e.g., a satellite image.

In some embodiments, correlating a 3D representation with a top view may include projecting the 3D representation on the top view. For example, the 3D representation of the street may be projected on, aligned (or registered) with a satellite image or a road or other map in database131. A map or image in database131with which a 3D representation may be correlated as described may include, or may be associated with, metadata related to elements, features or objects, e.g., metadata included in element representations132. For example, the location (e.g., in the form of coordinate values in a global coordinate system) of elements in a map may be known. Accordingly, by correlating elements in a 3D representation with a map or other data in database131, an embodiment may determine the exact location and/or orientation of the space in a coordinate system. For example, by projecting the 3D representation of the street on a map and aligning the 3D projection with the map, the location and orientation of the street may be determined.

As described, an embodiment may determine a location and/or orientation of a sensor in a space. In some embodiments, having determined the exact location and/or orientation of a space in a coordinate system as described, the exact location and/or orientation of a sensor in the space may be determined (e.g., by identifying elements in the space as described). For example, if the relative location of sensor311with respect to a number of elements in a space (e.g., house321, pole323etc.) is determined as described and the location and orientation of the space are determined as described (e.g., by registering or projecting a 3D object on a map) then an embodiment may readily determine the location and/or orientation of sensor311in any coordinate system. Accordingly, to determine a location and/or orientation of a sensor, an embodiment may capture images around the sensor, generate a 3D representation of the space surrounding the sensor, determine the exact location and orientation of the space in a global coordinate system and, based on the (possibly relative) location and/or orientation of the sensor in the space, determine the location and/or orientation of the sensor in the global coordinate system. For example, while a vehicle is traveling through a street, an embodiment may capture images (or video footage) in the street, create a 3D representation of the street, determine the exact location and/or orientation of the street (e.g., in a global or known reference such as a map) and, based on the location and/or orientation of the vehicle in the street, determine the exact location and/or orientation of the vehicle with respect to a map or any global or common reference of choice.

In some embodiments, a 3D digital representation may be created based on a set of images and based on a relative location of at least one element in at least some of the images. For example, devices or systems such as an odometer, Inertial Measurement Unit (IMU) simultaneous localization and mapping (SLAM) system may be used to determine a relative point of view (or location and/or orientation) for some or even each of a set of images obtained as a camera or other sensor is moving through a space, e.g., traveling in a street. Using the relative point of view of a set of images, a 3D digital representation of a space may be created.

Although, for the sake of clarity and simplicity, a camera and images are mainly discussed herein, it will be understood that input from any sensor or modality may be used for determining a location and orientation of a space by correlating a 3D representation with a top view representation of the space. For example, instead of, or in addition to, a camera as described, a LIDAR or imaging radar system may be used for creating a 3D representation.

In some embodiments, a top view representation of a space includes at least one of: an aerial image, a satellite image, a structural map, a road map and a DEM. Any information or data may be used to create a top view digital representation without departing from the scope of the invention.

In some embodiments, a portion of a top view representation may be selected based on location information of the space. For example, using GPS or other means, a location or an estimation of a location of sensor311may be determined and a portion of a satellite image covering the area of sensor311may be selected for the projection of a 3D digital representation as described, such that speed and efficiency are increased and/or amount of required memory or computational resources is reduced.

As described, embodiments of the invention provide a number of advantages over known or current systems and methods. For example, a system and method may autonomously and/or automatically identify (and add to a database representations of) elements, objects or features. Moreover, a system and methods may identify (and add to a database representations of) elements, objects or features that would not be naturally, or instinctively, identified by a human. For example, to guide a friend coming to visit, one might tell the friend “make a left right after the big oak tree” since a tree is something readily identified by humans. In contrast, a system and method may identify a part of the oak's trunk as an element and add a representation of the identified part of the trunk to DB217. For example, a part of a tree trunk identified by a LIDAR system may be an element represented by an element representation132since it is readily and/or accurately identified by a LIDAR system while a camera or human would have a difficulty identifying such element. For example, a human would not be able to identify a tree with a trunk that is two feet in diameter, however, an embodiment may identify the trunk's diameter with great precision, represent the trunk in DB217and consequently use the tree trunk for determining a location as described.

Since embodiments of the invention may identify, define (and represent in a database, e.g., in element representations132) elements, objects or features in space based on any sensor or modality, embodiments of the invention can identify (and represent in a database and use to determine a location as described) elements, objects or features that cannot be identified by known systems and methods nor by humans. Otherwise described, while known systems and methods (and humans) rely on objects or elements such as houses or road signs, embodiments of the invention create a view of a space based on the way that any sensor or modality “sees” or “senses” space. To illustrate, the sound of a power generator picked up by a microphone cannot be used by any known system or method to determine a location, however, for some embodiments of the invention, the sound of the generator (e.g., its audio characteristics) is a perfectly legitimate feature or element that may be represented by an element representation132and used for determining a location as described. By creating a view of space based on how it is sensed by a variety of sensors, embodiments of the invention provide an innovative way of representing space and navigating therein.

Using machine learning or deep learning, embodiments of the invention may create a representation of an element, object or feature based on input from a first type of sensor or modality and then identify the element, object or feature based on input from a second, different type of sensor or modality. For example, using machine or deep learning, an embodiment may know, or predict the input that a LIDAR system will provide when sensing a tree trunk as described. Accordingly, provided with an image of a tree trunk, an embodiment may accurately predict the input that a LIDAR system will provide when traveling near the tree. For example, based on an image from a camera, DB217may include an element representation132describing a pole323, based on a mapping330created as described, an embodiment may know, or predict, the input that a LIDAR system will provide when sensing (or traveling near) pole323.

Accordingly, embodiments of the invention can create a representation of an element in a space based on a first type of sensor and then identify the element based on input from a second, different type of sensor without having to collect additional data, e.g., an image of pole323and a mapping330that translate elements in an image to input from a LIDAR system may suffice for OLO233in order to identify pole323based on input from a LIDAR device even though this may be the first time input from a LIDAR that senses pole323is ever received. Otherwise described, an embodiment may create an element representation132for an object in space based on input from a first sensor type (e.g., a camera) and then identify the object based on input from a second, different sensor type even if the object has never before been sensed by a sensor of the second type.

Mapping330may be, may be included in, or may be created based on, a model created using machine or deep learning or NN. Mapping330may create, define or include a mapping or link between two or more sensor types or sources of information with respect to objects, elements or features in space. Machine learning may be used, by embodiments of the invention in order to understand the connection between two or more sources of information. For example, machine learning may be used to associate input from a LIDAR system to an element representation132where the element representation132was created based on an image. For example, provided with input from a LIDAR when sensing houses321, a system may train a model usable for translating input from LIDAR sensors such that the input is mapped to element representations related to houses321. Otherwise described, a model may link input from LIDAR systems to input from a camera in a way that enables creating element representations132of houses based on images and then determining that an element sensed by a LIDAR is one that is represented by an element representation132created based on an image. Of course, LIDAR and camera are only some examples of modalities brought here as examples

For example, mapping330may map input from a camera to input from a LIDAR system such that based on an image of objects such as pole323and features such as color change322, input from a LIDAR system can be mapped to these objects (e.g. the input may be mapped to an element representation132), thus, provided with an element representation132of pole323created based on an image of pole323(and based on data in DB232), OLO233can determine presence of pole323based on input from a LIDAR system. Generally, element representations132may describe elements or objects which are fixed in time and space and using mapping330as described, these elements may be identified using input from any sensor. As described, element representations132may be automatically created based on input from any sensors and, as described, element representations132may describe elements or features that are not necessarily ones that are native (or even visible) to a human or camera, rather, element representations132may be created based on input from any sensor or modality.

It will be understood that once a mapping330was created for a specific modality or sensor type with respect to an object or object type, the object or object type (or instances of the object type) may be identified in any place, location or space. For example, a system may be trained to identify poles323based on input from a LIDAR sensing poles323in a first town or region (first location) and, once the training is complete, input a LIDAR system in another city or region (second location) may be identified as related to poles323or it may be used to determine presence of poles323. For example, at a first step, training of a model (or creating a mapping) may be done such that input from a LIDAR system can be readily mapped to element representations132of poles, e.g., based on input from a LIDAR system in a first city, region or part of the world and based on knowledge of presence of poles in the region. At a next step, when mobile device230is located (or traveling) in another city or region, element representations132describing objects in the second city or region are downloaded to DB232and the model or mapping are also downloaded to DB232. Next, using input from a LIDAR device in mobile device230and the model or mapping, OLO233may identify poles in the second city, e.g., identify, in DB232, element representations132of nearby poles. It will be noted that OLO233needs not know that the identified objects are poles, all OLO233needs to determine is that input from a LIDAR system corresponds to an element representation132in DB232, as described, this correspondence may be achieved based on a model that maps input from the LIDAR system to an element representation132. For example, when mobile device230is a vehicle driven in Paris then server210may download to DB232element representations132that describe (or that are related to) elements, features or objects in Paris, using a model or mapping330as described, OLO233may map input from a LIDAR or any other sensor type to element representations132in DB232even though the mapping or model were created based on input from a LIDAR system located in London, of course, at least some element representations132related to objects in Paris need to be included in DB217(and downloaded to DB232) such that they can be found by OLO233when traveling in Paris. For example, server210may update DB217and/or dataset215based on input from sensors from anywhere in the world such that element representations132describing objects anywhere in the world are included in DB217and/or dataset215and thus the relevant element representations132can be downloaded to a vehicle (mobile device230) based on a region or area where mobile device230is located.

Accordingly and as described, some embodiments of the invention may include in a database representations of elements in a region or space based on input from a first sensor type or modality (e.g., a camera) and subsequently map input from a second, different sensor type or modality (e.g., a LIDAR) to the representations, accordingly, when a new sensor type or modality is added, embodiments of the invention do not need to collect information from the new sensor type in order to use input from the new sensor type, rather, input from the new sensor type may be mapped to existing element representations already included in a database. This capability greatly improves the field since it enables introducing new sensor types automatically, without having to first collect information from a new sensor. As described, using machine learning, a mapping330between input from a first type of sensor to input from a second type of sensor is created such that given input from the first sensor type when sensing an element an embodiment can map input from a second type of sensor to a representation of the element. For example, system200may automatically create a new element representation132object based on an image (first sensor type) of pole232and, using mapping330(or a model as described) map input from a LIDAR (second sensor type, when it senses pole323) to the element representation132object that describes pole323. By mapping input from a plurality of different sensor types or modalities to an element representation132object embodiments of the invention enable using any sensor type or modality to identify elements, objects or features in a space.

As further described, element representation132objects may be automatically created based on input from various or different sensors or modalities. An embodiment may continuously and automatically update element representation132objects, determine which element representations132are best used by different sensors or under different conditions. For example, if an embodiment determines that the color change feature322is not always identified then the embodiment may select to remove this feature from a databases or select not to use this feature, in another example, if an embodiment sees that house321is always sensed by a camera and a LIDAR then the embodiment may select to keep and use this object as described. Accordingly, embodiments of the invention may automatically and autonomously, without intervention or supervision of a human, create representations of elements in space and use the representations to determine a location as described.

In the description and claims of the present application, each of the verbs, “comprise” “include” and “have”, and conjugates thereof, are used to indicate that the object or objects of the verb are not necessarily a complete listing of components, elements or parts of the subject or subjects of the verb. Unless otherwise stated, adjectives such as “substantially” and “about” modifying a condition or relationship characteristic of a feature or features of an embodiment of the disclosure, are understood to mean that the condition or characteristic is defined to within tolerances that are acceptable for operation of an embodiment as described. In addition, the word “or” is considered to be the inclusive “or” rather than the exclusive or, and indicates at least one of, or any combination of items it conjoins.

Descriptions of embodiments of the invention in the present application are provided by way of example and are not intended to limit the scope of the invention. The described embodiments comprise different features, not all of which are required in all embodiments. Some embodiments utilize only some of the features or possible combinations of the features. Variations of embodiments of the invention that are described, and embodiments comprising different combinations of features noted in the described embodiments, will occur to a person having ordinary skill in the art. The scope of the invention is limited only by the claims.