Navigation system with mono-camera based traffic sign tracking and positioning mechanism and method of operation thereof

A method of operation of a navigation system comprising: receiving multiple frames, including a first frame and a second frame, of images; detecting a traffic sign from the images between the first frame and the second frame based on a sign recognition model; extracting a first image from the first frame and a second image from the second frame; matching the traffic sign is the same in the first image and the second image based on a similarity model; generating a sign location of the traffic sign with an inertial measurement unit reading based on the first image and the second image; and generating a global coordinate for the sign location for displaying on a navigation map.

TECHNICAL FIELD

An embodiment of the present invention relates generally to a navigation system, and more particularly to a system for mono-camera based traffic sign tracking and positioning mechanism.

BACKGROUND

As users become more empowered with the growth of navigation devices and vehicle based navigation services, new and old paradigms begin to take advantage of this new device space. There are many technological solutions to take advantage of this new device capability to enhance or augment navigation and to improve vehicle safety. However, users are often not provided with the ability to determine the position of traffic signs during navigation or route guidance. Research and development in the existing technologies can take a myriad of different directions.

Thus, a need still remains for a navigation system with a mono-camera based traffic sign tracking and position mechanism for operator awareness while using a navigation system and route guidance. In view of the ever-increasing commercial competitive pressures, along with growing consumer expectations and the diminishing opportunities for meaningful product differentiation in the marketplace, it is increasingly critical that answers be found to these problems. Additionally, the need to reduce costs, improve efficiencies and performance, and meet competitive pressures adds an even greater urgency to the critical necessity for finding answers to these problems.

SUMMARY

An embodiment of the present invention provides a method of operation of a navigation system including: receiving multiple frames, including a first frame and a second frame, of images; detecting a traffic sign from the images between the first frame and the second frame based on a sign recognition model; extracting a first image from the first frame and a second image for the second frame; matching the traffic sign is the same in the first image and the second image based on a similarity model; generating a sign location of the traffic sign with an inertial measurement unit reading based on the first image and the second image; and generating a global coordinate for the sign location for displaying on a navigation map.

An embodiment of the present invention provides a navigation system, including: a communication unit configured to receive multiple frames, including a first frame and a second frame, of images; and a control unit, coupled to the communication unit, configured to: detect a traffic sign from the images between the first frame and the second frame based on a sign recognition model; extract a first image from the first frame and a second image for the second frame; match the traffic sign is the same in the first image and the second image based on a similarity model; generate a sign location of the traffic sign with an inertial measurement unit reading based on the first image and the second image; and generate a global coordinate for the sign location for displaying on a navigation map.

An embodiment of the present invention provides a non-transitory computer readable medium including instructions executable by a control circuit for a navigation system, the instructions including: receiving multiple frames, including a first frame and a second frame, of images; detecting a traffic sign from the images between the first frame and the second frame based on a sign recognition model; extracting a first image from the first frame and a second image for the second frame; matching the traffic sign is the same in the first image and the second image based on a similarity model; generating a sign location of the traffic sign with an inertial measurement unit reading based on the first image and the second image; and generating a global coordinate for the sign location for displaying on a navigation map.

DETAILED DESCRIPTION

In the following description, numerous specific details are given to provide a thorough understanding of the invention. However, it will be apparent that the invention may be practiced without these specific details. In order to avoid obscuring an embodiment of the present invention, some well-known circuits, system configurations, and process steps are not disclosed in detail.

The drawings showing embodiments of the system are semi-diagrammatic, and not to scale and, particularly, some of the dimensions are for the clarity of presentation and are shown exaggerated in the drawing figures. Similarly, although the views in the drawings for ease of description generally show similar orientations, this depiction in the figures is arbitrary for the most part. Generally, the invention can be operated in any orientation.

The term “module” referred to herein can include software, hardware, or a combination thereof in an embodiment of the present invention in accordance with the context in which the term is used. For example, the software can be machine code, firmware, embedded code, and application software. Also for example, the hardware can be circuitry, processor, computer, integrated circuit, integrated circuit cores, a pressure sensor, an inertial sensor, a microelectromechanical system (MEMS), passive devices, or a combination thereof. Further, if a module is written in the apparatus claims section below, the modules are deemed to include hardware circuitry for the purposes and the scope of apparatus claims.

The modules in the following description of the embodiments can be coupled to one other as described or as shown. The coupling can be direct or indirect without or with, respectively, intervening items between coupled items. The coupling can be physical contact or by communication between items.

Referring now toFIG.1, therein is shown a navigation system100with a mono-camera based traffic sign tracking and position mechanism in an embodiment of the present invention. The navigation system100includes a first device102, such as a client or a server, connected to a second device106, such as a client or server. The first device102can communicate with the second device106with a communication path104, such as a wireless or wired network.

For example, the first device102can be of any of a variety of computing devices, such as a cellular phone, a tablet computer, a smart phone, a notebook computer, vehicle embedded navigation system, or computing device. The first device102can couple, either directly or indirectly, to the communication path104to communicate with the second device106or can be a stand-alone device.

The second device106can be any of a variety of centralized or decentralized computing devices, sensor devices to take measurements or record environmental information, such as sensor instruments, sensor equipment, or a sensor array. For example, the second device106can be a multimedia computer, a laptop computer, a desktop computer, grid-computing resources, a virtualized computer resource, cloud computing resource, routers, switches, peer-to-peer distributed computing devices, or a combination thereof.

The second device106can be mounted externally or internally to a vehicle, centralized in a single room or within a vehicle, distributed across different rooms, distributed across different geographical locations, embedded within a telecommunications network. The second device106can couple with the communication path104to communicate with the first device102.

For illustrative purposes, the navigation system100is described with the second device106as a computing device, although it is understood that the second device106can be different types of devices, such as a standalone sensor or measurement device. Also for illustrative purposes, the navigation system100is shown with the second device106and the first device102as end points of the communication path104, although it is understood that the navigation system100can have a different partition between the first device102, the second device106, and the communication path104. For example, the first device102, the second device106, or a combination thereof can also function as part of the communication path104.

The communication path104can span and represent a variety of networks and network topologies. For example, the communication path104can include wireless communication, wired communication, optical, ultrasonic, or the combination thereof. Satellite communication, cellular communication, Bluetooth, Infrared Data Association standard (lrDA), wireless fidelity (WiFi), and worldwide interoperability for microwave access (WiMAX) are examples of wireless communication that can be included in the communication path104. Ethernet, digital subscriber line (DSL), fiber to the home (FTTH), and plain old telephone service (POTS) are examples of wired communication that can be included in the communication path104. Further, the communication path104can traverse a number of network topologies and distances. For example, the communication path104can include direct connection, personal area network (PAN), local area network (LAN), metropolitan area network (MAN), wide area network (WAN), or a combination thereof.

Referring now toFIG.2, therein is shown an example a top plan view of a vehicle202for the navigation system100ofFIG.1. As an example, the navigation system100can include or interact with the first device102ofFIG.1as the vehicle202. The vehicle202can also include one or more of environmental sensors210. The vehicle202is an object or a machine used for transporting people or goods. The vehicle202can also be capable of providing assistance in maneuvering or operating the object or the machine.

The vehicle202can include or represent different types of vehicles. For example, the vehicle202can be an electric vehicle, a combustion vehicle, or a hybrid vehicle. Also for example, the vehicle202can be an autonomous vehicle or non-autonomous vehicle. As a specific example, the vehicle202can include a car, a truck, a cart, or a combination thereof.

The vehicle202can include a device, a circuit, one or more specific sensors, or a combination thereof for providing assistance or additional information to control, maneuver, or operate the vehicle202. The vehicle202can include a vehicle communication circuit204, a vehicle control circuit206, a vehicle storage circuit208, other interfaces, or a combination thereof.

The vehicle storage circuit208can include a functional unit or circuit integral to the vehicle202and configured to store and recall information. The vehicle storage circuit208can be a volatile memory, a nonvolatile memory, an internal memory, an external memory, or a combination thereof. For example, the vehicle storage circuit208can be a non-volatile storage such as non-volatile random access memory (NVRAM), Flash memory, disk storage, or a volatile storage such as static random access memory (SRAM). The vehicle storage circuit208can store vehicle software, other relevant data, such as input information, information from sensors, processing results, information predetermined or preloaded by the compute system100or vehicle manufacturer, or a combination thereof.

The vehicle control circuit206can include a function unit or circuit integral to the vehicle202and configured to execute or implement instructions. The vehicle control circuit206can execute or implement the vehicle software to provide the intelligence of the vehicle202, the navigation system100, or a combination thereof. The request can be from other parts of the vehicle202, the navigation system100, or a combination thereof or external to the navigation system100.

The vehicle control circuit206can be implemented in a number of different manners. For example, the vehicle control circuit206can be a processor, an application specific integrated circuit (ASIC) an embedded processor, a microprocessor, a hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), or a combination thereof. As a more specific example, the vehicle control circuit206can include an engine control unit, one or more central processing unit, or a combination thereof.

The vehicle communication circuit204can include a function unit or circuit integral to the vehicle202and configured to enable external communication to and from the vehicle202. For example, the vehicle communication circuit204can permit the vehicle202to communicate with the first device102, the second device106ofFIG.1, the communication path104ofFIG.1, or a combination thereof. The vehicle communication circuit204can provide to other portions of the vehicle202, the navigation system100, or a combination thereof or external to the navigation system100.

The vehicle communication circuit204can also function as a communication hub allowing the vehicle202to function as part of the communication path104and not limited to be an end point or terminal circuit to the communication path104. The vehicle communication circuit204can include active and passive components, such as microelectronics or an antenna, for interaction with the communication path104. For example, the vehicle communication circuit204can include a modem, a transmitter, a receiver, a port, a connector, or a combination thereof for wired communication, wireless communication, or a combination thereof.

The vehicle communication circuit204can couple with the communication path104to send or receive information directly between the vehicle communication circuit204and the first device102, the second device106, or a combination thereof as endpoints of the communication, such as for direct line-of-sight communication or peer-to-peer communication. The vehicle communication circuit204can further couple with the communication path104to send or receive information through a server or another intermediate device in between endpoints of the communication.

The vehicle202can further include various interfaces. The vehicle202can include one or more interfaces for interaction or internal communication between functional units or circuits of the vehicle202. For example, the vehicle202can include one or more interfaces, such as drivers, firmware, wire connections or buses, protocols, or a combination thereof, for the vehicle storage circuit208, the vehicle control circuit206, or a combination thereof.

The vehicle202can further include one or more interfaces for interaction with an occupant, an operator or a driver, a passenger, or a combination thereof relative to the vehicle202. For example, the vehicle202can include a user interface including input or output devices or circuits, such as a screen or touch screen, a speaker, a microphone, a keyboard or other input devices, an instrument panel, or a combination thereof.

The vehicle202can further include one or more interfaces along with switches or actuators for physically controlling movable components of the vehicle202. For example, the vehicle202can include the one or more interfaces along with the controlling mechanisms to physically perform and control the maneuvering of the vehicle202, such as for automatic driving or maneuvering features.

The functional units or circuits in the vehicle202can work individually and independently of the other functional units or circuits. The vehicle202can work individually and independently from the first device102, the communication path104, the second device106, other devices or vehicles, or a combination thereof.

The functional units or circuits described above can be implemented in hardware. For example, one or more of the functional units or circuits can be implemented using the a gate, circuitry, a processor, a computer, integrated circuit, integrated circuit cores, a pressure sensor, an inertial sensor, a microelectromechanical system (MEMS), a passive device, a physical non-transitory memory medium containing instructions for performing the software function, a portion therein, or a combination thereof.

The environmental sensors210are each a device for detecting or identifying environment of the vehicle202. The environmental sensors210can detect, identify, determine, or a combination thereof for the vehicle202itself, such as for status or movement thereof. The environmental sensors210can detect, identify, determine, or a combination thereof for environment within a cabin of the vehicle202, an environment external to and surrounding the vehicle202, or a combination thereof.

For example, the environmental sensors210can include a location-movement sensor212, a visual sensor214, a radar sensor216, an accessory sensor218, a volume sensor220, or a combination thereof. The location-movement sensor212can identify or calculate a geographic location of the vehicle202, determine a movement of the vehicle202, or a combination thereof. Examples of the location-movement sensor212can include an accelerometer, a speedometer, a GPS receiver or device, a gyroscope or a compass, or a combination thereof. The vehicle202can include the environmental sensors210other than or in addition to the location-movement sensor212, such as thermal sensor. The thermal sensor can capture and provide temperature readings for portions of the vehicle202. The thermal sensor can also capture and provide temperature readings external to the vehicle202.

The visual sensor214can include a sensor for detecting or determining visual information representing the environment external to and surrounding the vehicle202. The visual sensor214can include a camera attached to or integral with the vehicle202. For example, the visual sensor214can include a camera, such as forward facing camera, a rear-view or back-up camera, a side-view or a blind-spot camera, or a combination thereof. Also for example, the visual sensor214can include an infrared sensor or a night vision sensor. As a specific example, the visual sensor214can include a mono-camera215. The mono-camera215detects or determines visual information utilizing a single vision system. The mono-camera215does not require a dual vision or stereo vision system.

The visual sensor214can further include a camera on the first device102connected to and interacting with the vehicle202. The visual sensor214can further include a cabin camera for detecting or determining visual information inside the vehicle or cabin of the vehicle.

The radar sensor216can include an object-detection system, device, or circuit. The radar sensor216can determine or identify an existence of an object or a target, such as an obstacle or another vehicle, external to the vehicle202a relative location or a distance between the object or the target and the vehicle202, or a combination thereof.

The radar sensor216can utilize radio waves to determine or identify an existence of the object or the target, the relative location or a distance from the vehicle202, or a combination thereof. For example, the radar sensor216can include a proximity sensor or warning system, such as for an area in front of, behind, adjacent to or on a side of, or a combination thereof geographically or physically relative to the vehicle202.

The accessory sensor218can include a sensor for determining or detecting a status of a subsystem or a feature of the vehicle202. The accessory sensor218can determine or detect the status or a setting for windshield wipers, turn signals, gear setting, headlights, or a combination thereof.

The volume sensor220can include a sensor for detecting or determining sounds for the vehicle202. The volume sensor220can include a microphone for detecting or determining sounds within a cabin of the vehicle202. The volume sensor220can further include a circuit for detecting or determining a volume level or an output level of speakers within the vehicle202. The vehicle202can use one or more of the environmental sensors210describing or representing information regarding the environment within or surrounding the vehicle202.

The vehicle202can also include on-board diagnostics222(OBD) that can be accessed by the vehicle control circuit206. As an example, the vehicle control circuit206can access the on-board diagnostics222with the vehicle communication circuit204. The vehicle202can store and retrieve the on-board diagnostics222to and from the vehicle storage circuit208.

The on-board diagnostics222represent information about the vehicle202. For example, the on-board diagnostics222can provide status or the state of the vehicle202or a portion thereof. As a specific example, the on-board diagnostics222can represent an ignition status224.

The ignition status224represents the current state of the ignition. The ignition status224can represent whether the engine of the vehicle202is on or off. The term “on” refers to the state when the engine of the vehicle202is running. The term “off” refers to the state when the engine of the vehicle202is not running. The on-board diagnostics222can be carried over a vehicle bus226to communicate the ignition status224.

As an example, the vehicle bus226can be a controller area network (CAN) of the vehicle202that allows interaction between the vehicle communication circuit204, the vehicle control circuit206, the vehicle storage circuit208, the on-board diagnostics222, other interfaces, or a combination thereof.

The vehicle202can further include a user device or a mobile device illustrated inFIG.1. For example, the vehicle202can include the first device102. As a more specific example, the vehicle communication circuit204, the vehicle control circuit206, the vehicle storage circuit208, the environmental sensors210, one or more interfaces, or a combination thereof can be included in or make up the first device102included in or integral with the vehicle202. Also as a more specific example, the vehicle202can include or be integral with the first device102including an embedded computer system, an infotainment system, a smart driving or a driver assistance system, a self-driving or a maneuvering system for the vehicle, or a combination thereof.

Referring now toFIG.3, therein is shown an exemplary block diagram of the navigation system100. The navigation system100can include the first device102, the communication path104, and the second device106. The first device102can send information in a first device transmission308over the communication path104to the second device106. The second device106can send information in a second device transmission310over the communication path104to the first device102.

For illustrative purposes, the navigation system100is shown with the first device102as a client device, although it is understood that the navigation system100can have the first device102as a different type of device. For example, the first device102can be a server having a display interface.

Also for illustrative purposes, the navigation system100is shown with the second device106as a server, although it is understood that the navigation system100can have the second device106as a different type of device. For example, the second device106can be a client device.

For brevity of description in this embodiment of the present invention, the first device102will be described as a client device and the second device106will be described as a server device. The embodiment of the present invention is not limited to this selection for the type of devices. The selection is an example of an embodiment of the present invention.

The first device102can include a first control unit312, a first storage unit314, a first communication unit316, a first user interface318, and a first location unit320. The first control unit312can include a first control interface322. The first control unit312can execute a first software326to provide the intelligence of the navigation system100.

The first control unit312can be implemented in a number of different manners. For example, the first control unit312can be a processor, an application specific integrated circuit (ASIC) an embedded processor, a microprocessor, a hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), or a combination thereof. As a specific example, the first control unit312can represent the vehicle control circuit206ofFIG.2of the vehicle202ofFIG.2. The first control interface322can be used for communication between the first control unit312and other functional units in the first device102. The first control interface322can also be used for communication that is external to the first device102.

The first control interface322can be implemented in different ways and can include different implementations depending on which functional units or external units are being interfaced with the first control interface322. For example, the first control interface322can be implemented with a pressure sensor, an inertial sensor, a microelectromechanical system (MEMS), optical circuitry, waveguides, wireless circuitry, wireline circuitry, or a combination thereof.

The first location unit320can generate location information, current heading, and current speed of the first device102, as examples. The first location unit320can be implemented in many ways. For example, the first location unit320can function as at least a part of a global positioning system (GPS) such as a GPS receiver, an inertial navigation system, a cellular-tower location system, a pressure location system, or any combination thereof.

The first location unit320can include a first location interface332. The first location interface332can be used for communication between the first location unit320and other functional units in the first device102. The first location interface332can also be used for communication that is external to the first device102.

The first location interface332can include different implementations depending on which functional units or external units are being interfaced with the first location unit320. The first location interface332can be implemented with technologies and techniques similar to the implementation of the first control interface322.

The first storage unit314can store the first software326. The first storage unit314can also store the relevant information. For example, first storage unit314can store information such as a navigation map.

The first storage unit314can be a volatile memory, a nonvolatile memory, an internal memory, an external memory, or a combination thereof. For example, the first storage unit314can be a nonvolatile storage such as non-volatile random access memory (NVRAM), Flash memory, disk storage, or a volatile storage such as static random access memory (SRAM). As a specific example, the first storage unit314can represent the vehicle storage circuit208ofFIG.2of the vehicle202.

The first storage unit314can include a first storage interface324. The first storage interface324can be used for communication between and other functional units in the first device102. The first storage interface324can also be used for communication that is external to the first device102.

The first storage interface324can include different implementations depending on which functional units or external units are being interfaced with the first storage unit314. The first storage interface324can be implemented with technologies and techniques similar to the implementation of the first control interface322.

The first communication unit316can enable external communication to and from the first device102. For example, the first communication unit316can permit the first device102to communicate with the second device106, an attachment, such as a peripheral device or a computer desktop, and the communication path104.

The first communication unit316can also function as a communication hub allowing the first device102to function as part of the communication path104and not limited to be an end point or terminal unit to the communication path104. The first communication unit316can include active and passive components, such as microelectronics or an antenna, for interaction with the communication path104. As a specific example, the first communication unit316can represent the vehicle communication circuit204ofFIG.2of the vehicle202.

The first communication unit316can include a first communication interface328. The first communication interface328can be used for communication between the first communication unit316and other functional units in the first device102. The first communication interface328can receive information from the other functional units or can transmit information to the other functional units.

The first communication interface328can include different implementations depending on which functional units are being interfaced with the first communication unit316. The first communication interface328can be implemented with technologies and techniques similar to the implementation of the first control interface322.

The first user interface318allows a user (not shown) to interface and interact with the first device102. The first user interface318can include an input device and an output device. Examples of the input device of the first user interface318can include a keypad, a touchpad, soft-keys, a keyboard, a microphone, an infrared sensor for receiving remote signals, or any combination thereof to provide data and communication inputs. Examples of the output device can include a first display interface330and a first audio interface352.

The first display interface330can include a display, a projector, a video screen, a speaker, or any combination thereof. The first audio interface352can include sensors, speakers, microphones, headphones, subwoofers, surround sound components, transducers, or any combination thereof. The first display interface330and the first audio interface352allow a user of the navigation system100to interact with the navigation system100.

The first control unit312can operate the first user interface318to display information generated by the navigation system100. The first control unit312can also execute the first software326for the other functions of the navigation system100. The first control unit312can further execute the first software326for interaction with the communication path104via the first communication unit316.

The second device106can be optimized for implementing an embodiment of the present invention in a multiple device embodiment with the first device102. The second device106can provide the additional or higher performance processing power compared to the first device102. The second device106can include a second control unit334, a second communication unit336, and a second user interface338.

The second user interface338allows a user (not shown) to interface and interact with the second device106. The second user interface338can include an input device and an output device. Examples of the input device of the second user interface338can include a keypad, a touchpad, soft-keys, a keyboard, a microphone, or any combination thereof to provide data and communication inputs. Examples of the output device of the second user interface338can include a second display interface340and a second audio interface354.

The second display interface340can include a display, a projector, a video screen, a speaker, or any combination thereof. The second audio interface354can include sensors, speakers, microphones, headphones, subwoofers, surround sound components, transducers, or any combination thereof. The second display interface340and the second audio interface354allow a user of the navigation system100to interact with the navigation system100. The second control unit334can execute a second software342to provide the intelligence of the second device106of the navigation system100. The second software342can operate in conjunction with the first software326. The second control unit334can provide additional performance compared to the first control unit312.

The second control unit334can operate the second user interface338to display information. The second control unit334can also execute the second software342for the other functions of the navigation system100, including operating the second communication unit336to communicate with the first device102over the communication path104.

The second control unit334can be implemented in a number of different manners. For example, the second control unit334can be a processor, an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), or a combination thereof.

The second control unit334can include a second controller interface344. The second controller interface344can be used for communication between the second control unit334and other functional units in the second device106. The second controller interface344can also be used for communication that is external to the second device106.

The second controller interface344can be implemented in different ways and can include different implementations depending on which functional units or external units are being interfaced with the second controller interface344. For example, the second controller interface344can be implemented with a pressure sensor, an inertial sensor, a microelectromechanical system (MEMS), optical circuitry, waveguides, wireless circuitry, wireline circuitry, or a combination thereof.

A second storage unit346can store the second software342. The second storage unit346can also store the information for generating a map. The second storage unit346can be sized to provide the additional storage capacity to supplement the first storage unit314.

For illustrative purposes, the second storage unit346is shown as a single element, although it is understood that the second storage unit346can be a distribution of storage elements. Also for illustrative purposes, the navigation system100is shown with the second storage unit346as a single hierarchy storage system, although it is understood that the navigation system100can have the second storage unit346in a different configuration. For example, the second storage unit346can be formed with different storage technologies forming a memory hierarchal system including different levels of caching, main memory, rotating media, or off-line storage.

The second storage unit346can include a second storage interface348. The second storage interface348can be used for communication between other functional units in the second device106. The second storage interface348can also be used for communication that is external to the second device106.

The second storage interface348can include different implementations depending on which functional units or external units are being interfaced with the second storage unit346. The second storage interface348can be implemented with technologies and techniques similar to the implementation of the second controller interface344.

The second communication unit336can enable external communication to and from the second device106. For example, the second communication unit336can permit the second device106to communicate with the first device102over the communication path104.

The second communication unit336can also function as a communication hub allowing the second device106to function as part of the communication path104and not limited to be an end point or terminal unit to the communication path104. The second communication unit336can include active and passive components, such as microelectronics or an antenna, for interaction with the communication path104.

The second communication unit336can include a second communication interface350. The second communication interface350can be used for communication between the second communication unit336and other functional units in the second device106. The second communication interface350can receive information from the other functional units or can transmit information to the other functional units.

The second communication interface350can include different implementations depending on which functional units are being interfaced with the second communication unit336. The second communication interface350can be implemented with technologies and techniques similar to the implementation of the second controller interface344.

The first communication unit316can couple with the communication path104to send information to the second device106in the first device transmission308. The second device106can receive information in the second communication unit336from the first device transmission308of the communication path104.

The second communication unit336can couple with the communication path104to send information to the first device102in the second device transmission310. The first device102can receive information in the first communication unit316from the second device transmission310of the communication path104. The navigation system100can be executed by the first control unit312, the second control unit334, or a combination thereof. For illustrative purposes, the second device106is shown with the partition having the second user interface338, the second storage unit346, the second control unit334, and the second communication unit336, although it is understood that the second device106can have a different partition. For example, the second software342can be partitioned differently such that some or all of its function can be in the second control unit334and the second communication unit336. Also, the second device106can include other functional units not shown inFIG.3for clarity.

The functional units in the first device102can work individually and independently of the other functional units. The first device102can work individually and independently from the second device106and the communication path104.

The functional units in the second device106can work individually and independently of the other functional units. The second device106can work individually and independently from the first device102and the communication path104.

For illustrative purposes, the navigation system100is described by operation of the first device102and the second device106. It is understood that the first device102and the second device106can operate any of the modules and functions of the navigation system100.

Referring now toFIG.4, therein is shown an example of a display interface of the navigation system100ofFIG.1. The display interface can show a navigation map402. The navigation map402is a visual representation of a geographic area. For example, the navigation map402can be a representation of a state, a city, a town, a neighborhood, or any portion thereof. As a further example, the navigation map402can include the roadways in the geographic area. The navigation map402can be displayed by the first display interface330ofFIG.3, the second display interface340ofFIG.3, or a combination thereof. The navigation map402can be displayed in relation to a current location404.

The current location404can represent the physical location of a device or vehicle. As examples, the current location404can be generated by the location-movement sensor212ofFIG.2, the first location unit320ofFIG.3, or a combination thereof. Also as examples, the current location404can provide a geographic location of the first device102ofFIG.1, the vehicle202ofFIG.2, or a combination thereof.

The current location404can be generated in a number of ways. For example, the current location404can be determined by a global positioning system (GPS), cellular triangulation, wireless fidelity (WiFi) triangulation, dead reckoning, or a combination thereof. The navigation system100, the first device102ofFIG.1, the second device106ofFIG.1, the vehicle202, or a combination thereof can track or provide the current location404dynamically and in real time.

The navigation system100tracking, providing, or receiving the current location404“dynamically” refers to the receiving, monitoring, or a combination thereof of the current location404that is non-static or by potentially differing mechanism. The navigation system100tracking the current location404“real time” refers to the receiving, monitoring, or a combination thereof of the current location404at the time of reading for the current location404is taken regardless of the mechanism at some time period from the source of the reading of the current location404to the destination to receive the current location404without buffering delay for the purpose of storage and not for transmission. The navigation system100can display the current location404on the first display interface206, the second display interface234, or a combination thereof.

The navigation system100can display the current location404in a directionality406. The directionality406is the orientation towards movement. The directionality406can be determined by calculating the change in the current location404or based on dead reckoning, such as with an accelerometer or a gyroscope in the location-movement sensor212, first location unit320, or a combination thereof. For example, the directionality406can be determined based on the change in degrees of latitude, longitude, altitude, or a combination thereof, of the current location404.

The navigation system100can also display a current roadway408. The current roadway408is a road, path, a traversal route, or a portion thereof on which the vehicle202, the first device102, the second device106, or a combination thereof is currently travelling. For example, the current roadway408can be a street, an alleyway, a highway, a freeway, a parkway, an expressway, a toll road, a residential road, or an unpaved path. The current roadway408can be determined based on the current location404.

The navigation system100can display a travel route410for the vehicle202, the first device102, the second device106, or a combination thereof to navigate. The travel route410is the path to navigate from the current location404to a route destination412. For example, the travel route410can be determined based on the closest distance to the route destination412, fastest time of arrival to the route destination412, avoidance of tolls to the route destination412, or other constraints.

The route destination412refers to a geographic location or point of interest. The route destination412can include the end point or the termination of the route or the guidance. The route destination412can also include waypoints or intermediate stops. For example, the route destination412can be a store, a landmark, an office building or site, a park, an address, a general geographic area, a street, a city or municipality, or a combination thereof. Also for example, a waypoint for the route can represent the route destination412when the guidance is terminated at that particular instances of the waypoint.

For brevity of description, in this embodiment as an example, reference to the first display interface330and the first user interface318ofFIG.3is made, however, the descriptions with respect to the first display interface330and the first user interface318can be similarly applicable to the second display interface340and the second user interface338ofFIG.3.

In one embodiment, the first display interface330, in conjunction with the first user interface318, can enable a user of the navigation system100to input the route destination412to enable the navigation system100to generate the travel route410. The user can input the route destination412by using alpha-numeric characters, symbols, voice commands, gestures, or a combination thereof. For example, the user can input the route destination412by interfacing with the first user interface318, the second user interface338, or a combination thereof. As a further example, the user can input the route destination412by interfacing with the first audio interface352ofFIG.3, the second audio interface354ofFIG.3, or a combination thereof.

The navigation system100can process or utilize the current location404to obtain map information414. The map information414refers to a diagram or a collection of data representing an arrangement or distribution of geographic features, physical features, non-physical features, or a combination thereof of the geographic location on the navigation map402. For example, the map information414can include a physical feature such as a path, street name, an infrastructure, a geographical feature, a natural topology, points of interest, building, bodies of water, or a combination thereof. As a further example, the map information414can also include a non-physical feature such as a speed limit, a one-way designation, an address, points of interest (POI) or a combination thereof. As a specific example, the map information414can include a traffic sign416.

The traffic sign416provides instructions or information along the roadway. For example, the traffic sign416can include various types of signs controlling the flow of traffic, such as stop signs, yield signs, speed signs, one-way signs, no U-turn signs or a combination thereof. As further examples, the traffic sign416can include various types of signs providing information associated with the roadway, such as school zone signs, warning signs, pedestrian signs, or a combination thereof.

The navigation map402can display a sign location418of the traffic sign416. The sign location418is the geographic location of the traffic sign416in relation to the current roadway408. The sign location418can be displayed on the first display interface330, the second display interface340, or a combination thereof. For example, the sign location418can be on the left-hand side of the current roadway408, the right-hand side of the current roadway408, above the current roadway408, or a combination thereof. Also for example, the sign location418can be ahead or behind the current location404on the current roadway408.

Referring now toFIG.5, therein is shown an example of a first frame502with a first image504. The first frame502is an instance of a picture or segment of a video506captured by the mono-camera215ofFIG.2, the first device102ofFIG.1, the second device106ofFIG.1, or a combination thereof. The first frame502provides a visual representation of the view that is captured at a specific time by the mono-camera215, the first device102, the second device106, or a combination thereof.

The video506includes multiple frames that are captured over a period of time. As an example, the video506can include the first frame502. The video506provides a moving visual representation of the view of the mono-camera215, the first device102, the second device106, or a combination thereof. For example, the video506can consist of 30 instances of the frames per second. As a further example, the video506can consist of 60 instances of the frames per second. The video506can be displayed utilizing the first display interface330ofFIG.3, the second display interface340ofFIG.3, or a combination thereof.

Returning to the example, the first image504is an instance of an image generated for an object or area within in the first frame502. The first image504is the visual presentation of the view captured in the first frame502. For example, the first image504can be of pedestrians, cyclists, passenger cars, trucks, motorcycles, bicycles, roadways, building, or a combination thereof in the first frame502. As a specific example, the first image504can be of the traffic sign416in the first frame502. The first image504can be generated or extracted by the vehicle control circuit206ofFIG.2, the first control unit312ofFIG.3, the second control unit334ofFIG.3, or a combination thereof.

Referring now toFIG.6, therein is shown an example of a second frame602with a second image604. The second frame602is an instance of the frame in the video506captured by the mono-camera215ofFIG.2, the first device102ofFIG.3, the second device106ofFIG.4, or a combination thereof. The second frame602provides a visual representation of the view that is captured at a specific time by the mono-camera215, the first device102, the second device106, or a combination thereof. The second frame602can be an instance of the frame that was captured after or at a time after the first frame502ofFIG.5in the video506. For example, the second frame602can be the frame that follows milliseconds after the first frame502in the video506. As a further example, the second frame602can be the frame that follows seconds after the first frame502in the video506.

The second frame602can include the second image604. The second image604is an instance of the image generated for an object or area within the second frame602. For example, the second image604can be of pedestrians, cyclists, passenger cars, trucks, motorcycles, bicycles, or a combination thereof in the second frame602. As a specific example, the second image604can be of the traffic sign416in the second frame602. The second image604can be generated or extracted by the vehicle control circuit206ofFIG.2, the first control unit312ofFIG.3, the second control unit334ofFIG.3, or a combination thereof.

Referring now toFIG.7, therein is shown an exemplary control flow700of the navigation system100. The control flow700can be for tracking and locating the traffic sign416ofFIG.4with the navigation system100. The navigation system100can include a video capture module702, a sign recognition module704, a sign tracking module706, a sign extraction module708, a vehicle positioning module710, a global positioning module712, or a combination thereof.

The video capture module702can be coupled to the sign recognition module704. The sign recognition module704can be coupled to the sign tracking module706. The sign tracking module706can be coupled to the sign extraction module708. The sign extraction module708can be coupled to the vehicle positioning module710. The vehicle positioning module710can be coupled to the global positioning module712.

The modules can be coupled using wired or wireless connections, by including an output of one module as an input of the other module, by including operations of one module influence operation of the other module, or a combination thereof. The modules can be directly coupled with no intervening structures or objects other than the connector there-between, or indirectly coupled.

The video capture module702is configured to capture the video506ofFIG.5from the mono-camera215ofFIG.2, the first device102ofFIG.1, the second device106ofFIG.1, or a combination thereof. The video capture module702can capture the video506based on the ignition status224ofFIG.2of the vehicle202. The video capture module702can automatically capture the video506when the ignition status224is in the on position. The term “automatically” refers to when the mono-camera215, the first device102, the second device106, or a combination thereof captures the video215without a manual action for the specific purpose to capture the video215.

The video506can include frames714with images716. The frames714can include the first frame502ofFIG.5, the second frame602ofFIG.6, or a combination thereof. The usage of the terms “first” and “second” for the first frame502and the second frame602are a matter of convenience and are not to limit an order or importance. As an example, the first frame502and the second frame602can be swapped in the order when one was captured relative to the other. In other words, after the second frame602is captured, a subsequent frame can be captured and that subsequent frame can be termed as the first frame502.

The images716can include the first image504ofFIG.5, the second image604ofFIG.6, or a combination thereof. As with the first frame502and the second frame602, the usage of the terms “first” and “second” for the first image504and the second image604are a matter of convenience and are not to limit an order or importance. As an example, the first image504and the second image604can be swapped in the order when the first frame502and the second frame602are swapped in the order when one was captured relative to the other. In other words, after the second image604is captured in the second frame602, a subsequent image can be captured and that subsequent image can be termed as the first image504when the subsequent frame is termed as the first frame502.

The video capture module702can capture the video506, the frames714, the first frame502, the second frame602, the images716, the first image504, the second image604, or a combination thereof with the vehicle control circuit206ofFIG.2, the first control unit312ofFIG.3, the second control unit334ofFIG.3, or a combination thereof. The video capture module702can store the video506, the frames714, the first frame502, the second frame602, the images716, the first image504, the second image604, or a combination thereof in the vehicle storage circuit208ofFIG.2, the first storage unit314ofFIG.3, the second storage unit346ofFIG.3, or a combination thereof. The video capture module702can transmit the video506, the frames714, the first frame502, the second frame602, the images716, the first image504, the second image604, or a combination thereof with the vehicle communication circuit204ofFIG.2, the first communication unit316ofFIG.3, the second communication unit336ofFIG.3, or a combination thereof.

The control flow can pass from the video capture module702to the sign recognition module704. For example, the control flow can pass a processing result as an output from the video capture module702to an input of the sign recognition module704.

The sign recognition module704is configured to detect signs718in the video506, the frames714, the first frame502, the second frame602, or a combination thereof. The signs718can be any physical displays that provide information, including the traffic sign416ofFIG.4. The sign recognition module704can detect the signs718, the traffic sign416, or a combination thereof in the first frame502, the second frame602, or a combination thereof utilizing a sign detection model720.

The sign detection model720can be an artificial intelligence or machine learning implementation that can be trained to learn, determine, or detect the signs718from the video506. For example, the sign detection model720can be trained to learn to detect various instances of the signs718, including the types of the traffic sign416, from the video506, the frames714, images716in the frames714, or a combination thereof.

The sign detection model720can be trained utilizing information, from a database or other input with pictures of various instances of the signs718. For example, the sign detection model720can also detect the traffic sign414as one of the signs718. Also as examples, the information or pictures of the signs718can represent the traffic sign414at numerous angles, partial views, lighting, colors, types of signs, clarity, in various weather conditions, or a combination thereof. The sign detection model720can be trained and learn to detect various instances of the signs718even if the traffic sign414is not an exact example as the information or pictures used for training.

The sign detection model720can be implemented in a number of ways. For example, the sign detection model720can be implement with neural networks, such as convolution neural network, full connected network, or a combination thereof. Also for example, the sign detection model720can include unsupervised learning and other forms of supervised learning. The sign detection model720can work with artificial intelligence or machine learning that provides global minimum, one or more local minima, or a combination thereof.

Continuing with the example, the sign detection model720can detect various instances of the signs718from the video506, the frames714, the images716in the frames714, or a combination thereof. For example, the sign detection model720can also detect the traffic sign414as one of the signs718. As a specific example, the sign detection model720can detect the signs718in the first frame502, the second frame602, or a combination thereof. Also as a specific example, the sign detection model720can detect a stop sign as the traffic sign414in the first frame502, the second frame602, or a combination thereof. As a further example, the sign detection model720can detect a speed sign as the traffic sign414in the first frame502, the second frame602, or a combination thereof.

Continuing with the example, the sign detection model720can detect a number of the signs718. As a specific example, one of the signs718can include the traffic sign414. The sign detection model720can determine the instances of the traffic sign414from the number of the signs718.

The sign recognition module704, the sign detection model720, or a combination thereof can detect the signs718, the traffic sign414, or a combination thereof with the vehicle control circuit206, the first control unit312, the second control unit334, or a combination thereof. The sign recognition module704can transmit the signs718, the traffic sign414, or a combination thereof with the vehicle communication circuit204, the first communication unit316, the second communication unit336, or a combination thereof. The flow can progress to the sign tracking module706to process the images716containing the signs718, the traffic sign414, or a combination thereof detected by the sign detection model720.

The sign tracking module706can utilize the traffic sign414detected by the sign recognition module704to track each instance of the traffic sign414in the video506, the frames714, the images716in the frames714, or a combination thereof. Each instance of the traffic sign414can be included in a portion of the video506, the frames714, the images716of the frames714, or a combination thereof with no requirement to exist or be detected in the entirety of the video506or all of the frames714.

For example, the sign tracking module706can analyze the first frame502, the second frame602, or a combination thereof including each of the instances of the traffic sign414from the video506. The sign tracking module706can also analyze multiple instances of the first frame502, the second frame602, or a combination thereof. The sign tracking module706can capture the images716containing an instance of the traffic sign414from the first frame502, the second frame602, or a combination thereof.

The first frame502and the second frame602can be adjacent to each other and consecutive order of the frames714in the video506. The first frame502and the second frame602can also be nonadjacent to each other and not in consecutive order of the frames714in the video506. The location of the first frame502and the second frame602relative to each other in the frames714or within the video506is not limited to how far apart one is relative to the other.

As an example, the sign tracking module706detects each instance of the traffic sign414throughout the video506and across the frames714regardless of the location of the frames714within the video506. Also as an example, the sign tracking module706monitors and tracks each instance of the traffic sign414across the video506, the frames714, the images716of the frames714, or a combination thereof. The sign tracking module706can detect each instance of the traffic sign414independently, collectively, as subsets, or a combination thereof across the video506.

Continuing the example, the sign tracking module706can extract the images716of each of instance of the traffic sign414from the frames714. For example, the sign tracking module706can extract the first image504of the traffic sign414from the first frame502and the second image604of the traffic sign414from the second frame602. The sign tracking module706can track or match each instance of the traffic sign414in the first image504, the second image604, or a combination thereof utilizing a similarity model722.

The similarity model722measures the degree of similarity between each of instance of the traffic sign414between the first image504, the second image604, or a combination thereof. The similarity model722can determine whether the instances of the traffic sign414in the first image504, the second image604, or a combination thereof represent the same instance of the traffic sign414. For example, the similarity model722can utilize deep learning models to determine the degree of similarity.

The similarity model722can be implemented in a number of ways. For example, the similarity model722can be implement with neural networks, such as convolution neural network, full connected network, or a combination thereof. Also for example, the similarity model722can include unsupervised learning and other forms of supervised learning. The similarity model722can work with artificial intelligence or machine learning that provides global minimum, one or more local minima, or a combination thereof.

Similar to the sign detection model720, the similarity model722can be an artificial intelligence or machine learning implementation that can be trained to learn, determine, or detect types of the traffic sign414from the video506. For example, the similarity model722can be trained to learn to detect various instances of the traffic sign414from the video506, the frames714, images716in the frames714, or a combination thereof. The similarity model722can be trained utilizing information, from a database or other input with pictures of various instances of the traffic sign414. Also as examples, the information or pictures of the traffic sign414can represent the traffic sign414at numerous angles, partial views, lighting, colors, types of signs, clarity, in various weather conditions, or a combination thereof. The similarity model722can be trained and learn to detect the traffic sign414even if the traffic sign414is not an exact example as the information or pictures used for training.

Moreover, the similarity model722can be trained to learn to detect specific instances of the traffic sign414across the video506, the frames714, the images716of the frames714, or a combination thereof. In other words, the similarity model722can be utilized to match each specific instance of the traffic sign414in each of the frames714as needed and beyond a general detection of any instance or type of the traffic sign414.

The sign tracking module706can detect the traffic sign414in the first frame502, the second frame602, the first image504, the second image604, or a combination thereof with the vehicle control circuit206, the first control unit312, the second control unit334, or a combination thereof. The sign tracking module706can obtain the images716, the first image504, the second image604, or a combination thereof of the traffic sign414with the vehicle communication circuit204, the first communication unit316, the second communication unit336, or a combination thereof. The sign tracking module706can determine that the traffic sign414in the first frame502and the second frame602are the same instance of the traffic sign414with the vehicle control circuit206, the first control circuit312, the second control circuit334, or a combination thereof.

The control flow can pass to the sign extraction module708after tracking the traffic sign414in the video506, the frames714, the images716in the frames714, or a combination thereof. For example, the control flow can pass a processing result as an output from the sign tracking module706to an input of the sign extraction module708.

The sign extraction module708is configured to process each of the images716in the frames714including the first image504, the second image604, or a combination thereof. The sign extraction module708processes the first image504, the second image604, or a combination thereof by extracting features724representing or relating or associated with the signs718, the traffic sign414, or a combination thereof in the frames714, the first frame502, the second frame602, or a combination thereof. For example, the sign extraction module708can process the frames714, the first image504, the second image604, or a combination thereof utilizing the KAZE features. As a further example, the sign extraction module708can process the frames714, the first image504, the second image604, or a combination thereof utilizing ORB features.

The features724are the attributes or characteristics of or associated with the signs718or the traffic sign414in the first image504, the second image604, or a combination thereof. For example, the features724can include the shape of the signs718, the traffic sign414, or a combination thereof. As a further example, the features724can include the wording, numbers, or a combination thereof of or associated with the signs718, the traffic sign414, or a combination thereof. Further for example, the features724can also include symbols, such as arrows, lines, outlines of animals, or a combination thereof.

The sign extraction module708can determine the features724associated with the signs718, the traffic sign414, or a combination thereof from the frames714and the images716in the frames714. The sign extraction module708can determine the features724associated with the signs718, the traffic sign414, or a combination thereof based on information not directly from the frames714or the images714but from other information associated with the images716.

As an example, the sign extraction module708can utilize geolocation tagged with the frames714to pull up map data for confirmation of existence or location of the signs718, the traffic sign414, or a combination thereof. As a further example, the sign extraction module708can determine the features724associated with the existence or location of the signs718, the traffic sign414, or a combination thereof, such as an intersection, railroad tracks, construction work along roadway, or a combination thereof. The sign extraction module708can determine the existence or location of the signs718, the traffic sign414, or a combination thereof from external sources, such as intersection or railroad crossing from map data that is often associated with the signs718.

The sign extraction module708can determine the features724associated with the signs718, the traffic sign414, or a combination thereof between the frames714and the images716in the frames714. For example, the sign extraction module708can determine the same instance of the features724for the traffic sign414in the first image504of the first frame502, the second image604of the second frame602, or a combination thereof. As a further example, the sign extraction706can determine the same instance of the features724for the sign414in the first image504of the first frame502, the second image604of the second frame602, or a combination thereof.

The sign extraction module708can process the first image504, the second image604, or a combination thereof with the vehicle control circuit206, the first control unit312, the second control unit334, or a combination thereof. The sign extraction module708can communicate the features724of the traffic sign414with the vehicle communication circuit204, the first communication unit316, the second communication unit336, or a combination thereof. The sign extraction module708can store the features724of the traffic sign414with the vehicle storage circuit208, the first storage unit314, the second storage unit346, or a combination thereof.

The control flow can pass to the vehicle positioning module710when the sign extraction module708determines features724of the traffic sign414are found in the first image504and the second image604. The vehicle positioning module710can utilize the traffic sign414found in the first image504, the second image604, or a combination thereof.

The vehicle position module710is configured to determine a vehicle relative position726based on the first frame502, the second frame602, or a combination thereof. The vehicle relative position726is the location of the vehicle202in relation to the traffic sign414in the first frame502, the second frame602, or a combination thereof. The vehicle position module710measures the movement of the vehicle202based on the features724of the traffic sign414in the first frame502, the second frame602, or a combination thereof. For example, the vehicle positioning module710can determine the vehicle relative position726based on an inertial measurement unit reading728, a camera compensation730, or a combination thereof.

The inertial measurement unit reading728is the measurement of the translation and rotation of the vehicle202between the first frame502and the second frame602. The inertial measurement unit reading728can obtain the translation and rotation of the vehicle202from the location-movement sensor212ofFIG.2, the first location unit320ofFIG.3, or a combination thereof. For example, the inertial measurement unit reading728can provide the movement of the vehicle202based on the accelerometer, gyroscope, or a combination thereof. As a further example, the inertial measurement unit reading728can be synchronized with the mono-camera215, the first device102, the second device106, or a combination thereof to obtain the translation and rotation of the vehicle202between the first frame502, the second frame602, or a combination thereof.

The camera compensation730is the adjustment calculated to determine the accurate distance and depth of objects captured in the images716. The vehicle positioning module710generates the camera compensation730based on the parameters of the camera capturing the first frame502, the second frame602, or a combination thereof. For example, the camera compensation730is calculated for the mono-camera215, the first device102, the second device106, or a combination thereof.

The camera compensation730can allow for the correct determination of the distance and depth of the traffic sign414based on the first image504, the second image604, or a combination thereof. As a specific example, the vehicle positioning module710can generate the camera compensation730for the mono-camera215to determine the correct distance and depth of the traffic sign414in the first image504, the second image604, or a combination thereof.

The vehicle positioning module710can calculate the vehicle relative position726, the camera compensation730, or a combination thereof with the vehicle control circuit206, the first control unit312, the second control unit334, or a combination thereof. The vehicle positioning module710can obtain the inertial measurement unit reading728with the vehicle communication circuit204, the first communication unit316, the second communication unit336, or a combination thereof. The vehicle positioning module710can store the vehicle relative position726, the inertial measurement unit reading728, the camera compensation730, or a combination thereof with the vehicle storage circuit208, the first storage circuit314, the second storage circuit346, or a combination thereof.

The control flow can pass to the global positioning module712after determining the vehicle relative position726based on the traffic sign414in the first frame502, the second frame602, or a combination thereof. For example, the control flow can pass a processing result as an output from the vehicle positioning module710to an input of the global positioning module712.

The three-dimensional coordinate734is the reconstruction of the traffic sign414, the vehicle202, or a combination thereof based on the vehicle relative position726, the sign location418, or a combination thereof. The three-dimensional coordinate734determines the distance between objects in three-dimensions. The three-dimensional coordinate734can be generated based on the vehicle relative position726, the sign location418, the features724of the traffic sign414, or a combination thereof between the first frame502and the second frame602. The three-dimensional coordinate734can calculate the distance of the traffic sign414by calculating the changes to the vehicle relative position726, the sign location418, the features724of the traffic sign414, or a combination thereof between the first frame502and the second frame602.

The global navigation satellite system reading736is a reading of the location-movement sensor212, the first location unit320, or a combination thereof. The global navigation satellite system reading736can provide the global coordinate732for the vehicle202, the first device102, the second device106, or a combination thereof. The global navigation satellite system reading736can be obtained for the first frame502, the second frame602, or a combination thereof.

The global positioning module712calculates the global coordinate732of the traffic sign414utilizing the distance between the vehicle202and the traffic sign414calculated utilizing the three-dimensional coordinate734and the global navigation satellite system reading736of the first frame502, the second frame602, or a combination thereof. The global positioning module712can display the traffic sign414on the navigation map402utilizing the global coordinate732of the traffic sign414.

The global positioning module712can calculate the three-dimensional coordinate734, the global coordinate732of the traffic sign414, or a combination thereof with the vehicle control circuit206, the first control unit312, the second control unit334, or a combination thereof. The global positioning module712can communicate the three-dimensional coordinate734, the global navigation satellite system reading736, or a combination thereof with the vehicle communication circuit204, the first communication unit316, the second communication unit336, or a combination thereof. The global positioning module712can display the global coordinate732of the traffic sign414on the navigation map402with the first display interface330, the second display interface340, or a combination thereof.

It has been discovered that the navigation system100provides improved operation safety of the vehicle202by displaying the traffic sign414, the sign location418, or a combination thereof on the navigation map402utilizing the global coordinates732of the traffic sign414to raise awareness to the user of the vehicle202.

It has further been discovered that the navigation system100can track and position the traffic sign414utilizing a cost-effective mono-camera215mechanism by calculating the global coordinate732of the traffic sign414utilizing the three-dimensional coordinate734, the global navigation satellite system reading736, or a combination thereof.

It has further been discovered that the navigation system100can verify the global coordinate732of the traffic sign414on the navigation map402utilizing the three-dimensional coordinate734, the global navigation satellite system reading736, or a combination thereof based on the video506, the frames714, the images716of the frames714, or a combination thereof. The navigation system100can further update the global coordinate732of the traffic sign414on the navigation map402based on the global coordinate732determined based on the three-dimensional coordinate734, the global navigation satellite system reading736, or a combination thereof.

The navigation system100has been described with module functions or order as an example. The navigation system100can partition the modules differently or order the modules differently. For example, the sign recognition module704can be coupled to the video capture module702. As a further example, the sign extraction module708can be coupled to the sign recognition module704.

For illustrative purposes, the various modules have been described as being specific to the first device102or the second device106. However, it is understood that the modules can be distributed differently. For example, the various modules can be implemented in a different device, or the functionalities of the modules can be distributed across multiple devices. Also as an example, the various modules can be stored in a non-transitory memory medium.

As a more specific example, one or more modules described above can be stored in the non-transitory memory medium for distribution to a different system, a different device, a different user, or a combination thereof, for manufacturing, or a combination thereof. Also as a more specific example, the modules described above can be implemented or stored using a single hardware unit, such as a chip or a processor, or across multiple hardware units.

The modules described in this application can be hardware implementation or hardware accelerators in the first control unit312or in the second control unit334. The modules can also be hardware implementation or hardware accelerators within the first device102or the second device106but outside of the first control unit312or the second control unit334, respectively, as depicted inFIG.3. However, it is understood that the first control unit312, the second control unit334, or a combination thereof can collectively refer to all hardware accelerators for the modules.

The modules described in this application can be implemented as instructions stored on a non-transitory computer readable medium to be executed by a first control unit312, the second control unit334, or a combination thereof. The non-transitory computer medium can include the first storage unit314, the second storage unit346, or a combination thereof. The non-transitory computer readable medium can include non-volatile memory, such as a hard disk drive, non-volatile random access memory (NVRAM), solid-state storage device (SSD), compact disk (CD), digital video disk (DVD), or universal serial bus (USB) flash memory devices. The non-transitory computer readable medium can be integrated as a part of the navigation system100or installed as a removable portion of the navigation system100.

The physical transformation from determining the global coordinate732of the traffic sign414results in the display of the traffic sign414in the physical world, such as display of the sign location418on the navigation map402. Movement in the physical world, such movement of the traffic sign414, results in changes to the sign location418by updating the navigation map302.

Referring now toFIG.8, therein is shown a flow chart of a method800of operation of a navigation system100in an embodiment of the present invention. The method800includes: receiving multiple frames, including a first frame and a second frame, of images in a block802; detecting a traffic sign from the images between the first frame and the second frame based on a sign recognition model in a block804; extracting a first image from the first frame and a second image from the second frame in a block806; matching the traffic sign is the same in the first image and the second image based on a similarity model in a block808; generating a sign location of the traffic sign with an inertial measurement unit reading based on the first image and the second image in a block810; and generating a global coordinate for the sign location for displaying on a navigation map in a block812.

The resulting method, process, apparatus, device, product, and/or system is straightforward, cost-effective, uncomplicated, highly versatile, accurate, sensitive, and effective, and can be implemented by adapting known components for ready, efficient, and economical manufacturing, application, and utilization. Another important aspect of an embodiment of the present invention is that it valuably supports and services the historical trend of reducing costs, simplifying systems, and increasing performance.

These and other valuable aspects of an embodiment of the present invention consequently further the state of the technology to at least the next level.