Patent Description:
Advanced driver assistance systems (ADAS) are systems intended to help the driver in the driving process, often aimed at increasing vehicle and road safety, as well as providing convenience. An example of an ADAS feature is lane departure warning. In certain conventional lane departure warning systems, one or more laser sensors, infrared sensors, or a combination thereof, can be implemented.

<CIT>) "Car control system uses infrared images to control vehicles head-lights and other units" may provide certain related disclosures in the field of information systems in vehicles.

<CIT> relates to an arrangement in which image data are detected from the surroundings. At least one region of a detected image is evaluated to form a characteristic measurement value. A visibility is determined using a reference relating the measurement value to a visibility. A measurement value for the sharpness or contrast of the image may be used, and obtained from the differences of intensities of adjacent or neighboring pixels. Independent claims are included for the following; an apparatus for estimating visibility; a system for acquiring references.

<CIT> relates to an apparatus or method for generating driver assistance information of a traveling vehicle. Traveling environments can be restructured into an image based on a sensor system capable of removing blind spots around a traveling vehicle, and a scheme capable of exactly detecting a lane and a vehicle, and give a collision warning when there is a possibility of collision. The apparatus or method integrates recognition results of front/rear vehicles based on a lane recognition method applicable to various road environments, including the shape, curvature and loss of a lane, improves the stability of lane recognition, and distinguishes a solid line from a dot line, so it is possible to provide a driver with the subdivided degrees of danger. In addition, when a vehicle deviates from a lane, a voice warning may be output and the position of the vehicle for a road is restructured as an image.

<CIT> relates to an image-sensing device for a driver assistance system having a mobile communications device which comprises an image sensor, as well as having a mounting device for mounting the communications device in a vehicle. The mounting device comprises a device adapter for holding the communications device, a vehicle adapter which is arranged fixed to the vehicle, and a universal module which is arranged between the vehicle adapter and the device adapter.

<CIT> relates to a vehicle is equipped with a camera (which may be a stereoscopic camera) and a computer for processing the image data acquired by the camera. The image acquired by the camera is processed by the computer, and features are extracted therefrom. The features are further processed by various techniques such as object detection/segmentation and object tracking/classification. The acquired images are sometimes contaminated by optical occlusions such as raindrops, stone-chippings and dirt on the windshield. In such a case, the occluded parts of the image are reconstructed by optical flow estimation or stereo disparity estimation. The fully reconstructed image is then used for intended applications.

None of <CIT>, <CIT>, <CIT>, <CIT> or <CIT> relate to an in-vehicle infotainment system.

Embodiments of the present invention are described more fully hereinafter with reference to the accompanying drawings in which embodiments of the present invention are shown.

Claimed embodiments herein are directed to augmenting ADAS features of a vehicle with image processing support in an on-board vehicle platform. A vehicle may include one or more processors, networking interfaces, and other computing devices that may enable it to capture image data, process the image data, and augment ADAS features of the vehicle with image processing support in an on-board vehicle platform. In some embodiments, images from the ADAS cameras may be transferred at a low frequency to an on-board vehicle platform, such as an in-vehicle infotainment (IVI) system. The on-board vehicle platform processes the images to offer additional ADAS features. The ADAS features implemented in the IVI system are typically not mission-critical safety features and thus would likely not need strict real-time performance. The IVI system is not a replacement to the dedicated ADAS processor but rather functions as a complimentary augmentation of the ADAS subsystem. In this regard, a computer-implemented method according to claim <NUM> and a system according to claim <NUM> are provided according to the present invention.

Preferred embodiments of the present invention are set forth in the dependent claims.

Example embodiments of the present invention will now be described with reference to the accompanying figures.

Referring now to <FIG>, illustrates an example system configuration <NUM>, in accordance with an embodiment of the present invention, for augmenting ADAS features of a vehicle with image processing support in on-board vehicle platform. The configuration includes one or more vehicles <NUM>. The vehicle <NUM> may include one or more systems that include one or more processing devices for implementing functions and features associated with the vehicle <NUM>, as will be discussed in greater detail below. The vehicle <NUM> includes one or more image sensors 106a-106b (collectively referred to as <NUM>) capable of capturing data associated with the vehicle <NUM>. For example, an image sensor <NUM> may be an external video camera 106a, 106b that may capture data, such as video, regarding road conditions as the vehicle <NUM> progresses in its trip.

The vehicle <NUM> includes a vehicle on-board platform, such as an in-vehicle infotainment (IVI) system <NUM>. As used herein, an IVI system <NUM> may refer to a system in a vehicle <NUM> that provides entertainment and informational features for the vehicle <NUM>. The IVI system <NUM> may be part of the vehicle's main computer or a stand-alone system. The IVI system <NUM> communicates with a system for augmenting ADAS features of a vehicle <NUM>, as described herein. The IVI system <NUM> may further include one or more processors communicatively coupled to an electronic memory, described in greater detail below.

The vehicle <NUM> may establish a connection with a remote server <NUM> over multiple types of networks <NUM>, such as a wireless fidelity (Wi-Fi) network, a Wi-Fi Direct network, Bluetooth®, a radio network, a cellular network (e.g., third generation or fourth generation), a satellite network, a cable network, a landline-based network, the Internet, intranets, a telephone network, a television network, data networks, or other communication mediums connecting multiple computing devices to one another, as non-limiting examples. According to certain embodiments herein, multiple networks may be leveraged by the vehicle <NUM> to enable communication with the remote server <NUM>.

In some embodiments, the vehicle <NUM> may be configured to be coupled to an electronic device <NUM>. The electronic device <NUM> may include one or more electronic device processors communicatively coupled to an electronic device memory, as well as user interface and an output element, such as a speaker of the vehicle <NUM>. The electronic device <NUM> may communicate with the vehicle <NUM> via a communicative link. In certain embodiments herein, devices related to the implementation of augmenting ADAS features of a vehicle <NUM> may exist onboard an IVI system <NUM> such that the functionality described herein may be associated with the IVI system <NUM>. In other embodiments, the functionality described herein may reside independently of other systems or may be associated with various other systems.

The IVI system <NUM> may be in communication with one or more electronic devices <NUM>. In one aspect, an electronic device <NUM> may serve as an extension of the IVI system <NUM>. For example, if the IVI system <NUM> does not have Internet capabilities, the IVI system <NUM> may communicate with an electronic device <NUM> associated with the vehicle <NUM> to utilize the communication capabilities of the electronic device <NUM>.

The communicative link may be any suitable electronic communication link including, but not limited to, a hardwired connection, a serial link, a parallel link, a wireless link, a Bluetooth® channel, a ZigBee® connection, a wireless fidelity (Wi-Fi) connection, a Near Field Communication (NFC) protocol, a proprietary protocol connection, or combinations thereof. In one aspect, the communicative link may be secure such that it is relatively difficult to intercept and decipher communications between the electronic device <NUM> and the IVI system <NUM>. In certain embodiments, the communicative link may be encrypted. Further, in certain embodiments, the communications may be encrypted at more than one open systems interconnections (OSI) model layer. For example, the communications between the electronic device <NUM> and the vehicle <NUM> may be encrypted at both the application layer and the transport or link layer. In some embodiments, the communicative link may be through the communication capabilities of an electronic device <NUM> associated with the vehicle <NUM>. For example, if the vehicle <NUM> does not have Internet capabilities, the IVI system <NUM> may be able to access data through its association with, for example, a smartphone with cellular communication capabilities.

For the purposes of this discussion, the vehicle <NUM> may include, but is not limited to, a car, a truck, a light-duty truck, a heavy-duty truck, a pickup truck, a minivan, a crossover vehicle, a van, a commercial vehicle, a private vehicle, a sports utility vehicle, a tractor-trailer, an aircraft, an airplane, a jet, a helicopter, a space vehicle, a watercraft, a motorcycle, or any other suitable vehicle with information and media capability. However, it will be appreciated that embodiments of the disclosure may also be utilized in other transportation or non-transportation related application where augmenting ADAS features may be implemented.

Vehicle sensors may be any suitable data-gathering element associated with the vehicle <NUM>. As a result, vehicle sensors may gather audio, visual, tactile, or environmental information within or associated with the vehicle <NUM>. For example, the vehicle sensors may include one or more cameras 106c in the cabin of the vehicle <NUM> that may capture images of occupants as well as scene information, such as lighting conditions within the vehicle <NUM> or weather outside of the vehicle <NUM>. Vehicle sensors may also be GPS devices, a microphone, seat weight sensors, or other type of data-gathering element associate with the vehicle <NUM>.

Furthermore, a suitable electronic device can include, but are not limited to, smartphones, tablets, laptop computers, electronic book reading devices, processor-based devices, etc..

<FIG> depicts a block diagram of an example vehicle computing system <NUM> in a vehicle, e.g., vehicle <NUM> in <FIG>, for augmenting ADAS features of a vehicle <NUM> with image processing support in an IVI system <NUM>, among other things. As shown in <FIG>, multiple vehicle systems interact to facilitate providing augmented ADAS features. For example, a computing system <NUM> exists for controlling a vehicle's standard devices or components, which may include engine devices, braking devices, power steering devices, door control devices, window control devices, etc. The computing system <NUM> also includes various input/output ("I/O") devices <NUM> that may exist in a vehicle <NUM>, such as collection devices, such as vehicle sensors (e.g., a microphone, a seat weight sensors, cameras, both interior-facing cameras 106c for capturing images within a vehicle <NUM> and exterior-facing cameras 106a, 106b for capturing images from a vehicle's surroundings) and display devices, such as light-emitting diode ("LED") displays and organic light-emitting diode ("OLED") displays, as non-limiting examples. A main processor <NUM> communicates with the standard engine control devices <NUM> and I/O devices <NUM> to activate the devices, send information to these devices, or collect information from these devices, as non-limiting examples.

The computing system <NUM> includes single-feature fixed-function devices, such as an ADAS image System on Chip (SoC) <NUM>. An SoC may be an integrated circuit that integrates all components of a computing system into a single chip. The SoC may contain digital, analog, mixed-signal, and/or radio-frequency functions that may all be on a single chip substrate. The SoC may include a microcontroller, microprocessor, or digital signal processor core(s). In some embodiments, SoCs may include more than one processor core. SoCs may further include blocks, which may include a ROM, RAM, Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, and/or any other type of non-volatile memory. SoCs may include one or more timing sources, such as oscillators and/or phase-locked loops. SoCs may include interfaces, such as USB, FireWire, Ethernet, Universal Asynchronous Receiver/Transmitter (USART), Serial Peripheral Interface (SPI) bus, and/or Media Oriented Systems Transport (MOST) bus. In some embodiments, the ADAS image SoC <NUM> may communicate with one or more I/O devices <NUM> of the computing system <NUM>.

The computing system <NUM> is in communication with the IVI system <NUM>. As used herein, an IVI system prefers to a system in a vehicle that provides entertainment and informational features for the vehicle.

The IVI system <NUM> includes, but is not limited to, a processor <NUM>, a memory <NUM>, one or more communication devices <NUM>, and a transceiver <NUM>. The processor <NUM> communicates with the communication devices <NUM> in the IVI system <NUM>. For example, the processor <NUM> may communicate with the memory <NUM> to execute certain computer-executable instructions or modules, such as <NUM>, <NUM>, <NUM>, <NUM>, stored in the memory <NUM> to facilitate augmenting ADAS features of the vehicle <NUM> as described herein. The processor <NUM> also communicates with the one or more communication devices <NUM> to send and receive messages from various types of networks, such as those listed above. A transceiver <NUM> facilitates the sending and receipt of such messages. In some embodiments, a transmitter and a separate receiver may be utilized to send and receive messages, respectively.

The processor <NUM>, the memory <NUM>, the communication devices <NUM>, and the transceiver <NUM> are onboard a system board (hereinafter "onboard") in the IVI system <NUM>. In this way, these devices operate out of band, or with access to only minimal power, such as in association with a vehicle shutdown, hibernation, or standby, as non-limiting examples. In one example, a backup battery may be used to provide sufficient power to enable the devices in the IVI system <NUM> to operate out of band. Thus, the devices in the IVI system <NUM> remain awake (e.g., after a vehicle <NUM> has been shutdown) and provide certain functionality, such as communicating with a user device, e.g., electronic device, to send and receive messages in association with augmenting ADAS features of the vehicle <NUM>. Such functionality may be referred to herein as out of band or operating out of band. The devices in the IVI system <NUM> can also communicate with one another while operating out of band. For example, the processor <NUM> may communicate with the memory <NUM> to execute computer-executable instructions or modules therein while operating out of band.

The devices and/or program modules in the computing system <NUM> shut down when a vehicle <NUM> is powered down, for example, and therefore do not operate out of band. A main operating system (not shown) that may control standard components in a vehicle <NUM>, such as an engine, brakes, doors, windows, hard disks, or other devices in communication with the main operating system or one of its program modules, is not operational when the vehicle <NUM> is shut down. The O/S <NUM> in the memory <NUM>, however, is operational when the vehicle <NUM> is shut down, or otherwise in a low power state such as hibernation or standby, because it may be located onboard or at the board level in firmware. Such a configuration enables devices in the IVI system <NUM> to send messages, receive messages, and cause the performance of augmenting ADAS features of the vehicle <NUM>. As an example, according to certain embodiments, the processor <NUM> of the IVI system <NUM> may communicate with the main processor <NUM> (and/or other devices) of the computing system <NUM> to wake the main processor <NUM> so that it may cause performance of the functions requested by a user via one or more action codes. In one embodiment, such communication may occur via the CAN BUS protocol, as will be described in greater detail below.

In certain embodiments, the processor <NUM> of the IVI system <NUM> may also communicate with the main processor <NUM> and/or other devices of the computing system <NUM> in response to executing computer-executable instructions in a context engine, such as <NUM>, to generate or process context information.

The processors <NUM> and <NUM> may include any number of suitable processing devices, such as a central processing unit ("CPU"), a digital signal processor ("DSP"), a reduced instruction set computer ("RISC"), a complex instruction set computer ("CISC"), a microprocessor, a microcontroller, a field programmable gate array ("FPGA"), or any combination thereof. In one embodiment, the system <NUM> may be based on an Intel® Architecture system, and the processors <NUM> and chipset may be from a family of Intel® processors and chipsets, such as the Intel® Atom® processor family. The processor <NUM> may also include one or more processors as part of one or more application-specific integrated circuits ("ASICs") or application-specific standard products ("ASSPs") for handling specific data processing functions or tasks. Additionally, any number of suitable I/O interfaces and/or communications interfaces (e.g., network interfaces, data bus interfaces, etc.) may facilitate communication between the processors <NUM> and other components of the system <NUM>.

The one or more communication devices <NUM> facilitate communications between the system <NUM> and other devices that may be external to a vehicle <NUM> containing the system <NUM>. For example, the one or more communications devices <NUM> may enable the system <NUM> to receive messages from an electronic device <NUM> and/or send messages to an electronic device <NUM> as illustrated in <FIG>. The communication devices <NUM> may enable various types of communications over different networks, such as wireless networks including, but not limited to, a wireless fidelity (Wi-Fi) network, a Wi-Fi Direct network, a radio network, a cellular network, a GPS network, a ZigBee® connection, a Bluetooth® channel, proprietary protocol connections, and other wireless links, as well as hardwired connections, serial link connections, parallel link connections or combinations thereof.

According to various configurations, one or multiple interface cards or circuits may support the multiple networks named above. In one embodiment, such one or more interface cards or circuits may be onboard such that firmware in the memory <NUM> may access and control communications associated with the IVI system <NUM>.

Turning now to the contents of the memory <NUM>, the memory <NUM> may include any number of suitable memory devices, such as caches, read-only memory devices, random access memory ("RAM"), dynamic RAM ("DRAM"), static RAM ("SRAM"), synchronous dynamic RAM ("SDRAM"), double data rate ("DDR") SDRAM ("DDR-SDRAM"), RAM-BUS DRAM ("RDRAM"), flash memory devices, electrically erasable programmable read only memory ("EEPROM"), non-volatile RAM ("NVRAM"), universal serial bus ("USB") removable memory, magnetic storage devices, removable storage devices (e.g., memory cards, etc.), and/or non-removable storage devices. As desired, the memory <NUM> may include internal memory devices and/or external memory devices in communication with the system <NUM>.

The memory <NUM> may store data, executable instructions, and/or various program modules utilized by the processor <NUM>. Examples of data that may be stored by the memory <NUM> include data files <NUM> and any number of suitable program modules and/or applications that may be executed by the processor <NUM>, such as, but not limited to, an operating system ("OS") <NUM>, an vision analysis module <NUM>, an ADAS enhancing module <NUM>, a navigation engine <NUM>, and a context engine <NUM>. Each of these modules may be implemented as individual modules or, alternatively, one or more of the modules may perform all or at least some of the functionality associated with the other modules. In certain embodiments, these modules may be stored as firmware in a read-only memory <NUM>, thereby making it more difficult for the functions described herein to be tampered with or disabled.

The data files <NUM> may include any suitable information that may facilitate augmenting ADAS features of a vehicle <NUM>. Example information may include, but is not limited to, information that may be used to authenticate an electronic device <NUM>, capture data through one or more image sensors <NUM>, process images, and obtain information from a central server <NUM>, as well as other information that may facilitate the processes described herein.

The operating system <NUM> may include a suitable module or application that facilitates general operation of the system <NUM>, as well as the execution of other program modules illustrated in the memory <NUM> in <FIG>.

The vision analysis module <NUM> performs a number of functions to facilitate processing of images received from the ADAS image SoC <NUM>. The vision analysis module <NUM> is in communication with the ADAS enhancing module <NUM> to determine or identify types of processing to be applied to the received images.

The ADAS enhancing module <NUM> performs a number of functions to facilitate augmenting ADAS features of the vehicle. For instance, the ADAS enhancing module <NUM> receives or otherwise obtain information from one or more sources (e.g., navigation engine <NUM>, context engine <NUM>, remote server <NUM>, electronic device <NUM>, etc.). Further, ADAS enhancing module <NUM> communicates with vision analysis module <NUM> to process image using one or more applicable techniques or algorithms. ADAS enhancing module <NUM> then analyzes data generated during the processing of images and may determine or identify an action. For example, the action may be to dim the headlights, change the speed of the windshield wipers, or generate a warning to the driver of the vehicle to notify the driver of a lane departure by the vehicle <NUM>.

The navigation engine <NUM> may perform a number of functions to provide an electronic map combined with route instructions, usually displayed on a dashboard video screen. For example, navigation engine <NUM> may receive continuous information from one or more GPS devices of the vehicle and may update the location of the vehicle <NUM> on a map displayed to the driver. The navigation engine <NUM> may obtain information regarding retail businesses, restaurants, traffic conditions, and other points of interest based upon, at least in part, the data received from the one or more GPS devices of the vehicle <NUM>. The navigation engine <NUM> may communicate with the ADAS enhancing module <NUM> and transmit data to facilitate augmenting ADAS features of the vehicle <NUM>.

The context engine <NUM> may perform a number of functions to facilitate formation and processing of context information. For example, context engine <NUM> may identify existing context information based upon received data, generate context information based upon received data, or process (e.g., augment or update) context information based upon received data. The context engine <NUM> may obtain related information using the context information, such as recommendations or other information that may be used to assist the driver or occupant of the vehicle <NUM>. Context engine <NUM> may transmit the related information and/or the context information to the ADAS enhancing module <NUM> to facilitate augmenting ADAS features of the vehicle <NUM>.

One or more bus communication modules (not shown) may include various protocols that may be used by devices in the system <NUM> to communicate with one another. An example protocol may be the CAN (controller area network) BUS protocol, in which communication occurs between devices over a controller area network without a host computer device. For example, the processor <NUM> may use the CAN BUS protocol to communicate with a main processor <NUM> to wake the main processor <NUM> and instruct it to activate an I/O device <NUM>, in one example. Protocols in addition to the CAN BUS protocol, such as other message-based protocols, may be used in other embodiments. In other examples, a chipset (not shown) may be provided to control communications between the devices in the vehicle computing system <NUM>.

In addition to or alternative to the memory <NUM>, other embodiments may include one or more suitable computer-readable media that may be provided for storing computer-executable instructions such as those stored in the memory <NUM>. One or more processing devices, such as the processor <NUM>, may execute such computer-executable instructions to facilitate the remote management of a vehicle, as described above in association with the modules <NUM>, <NUM>, <NUM>, <NUM> in the memory <NUM>. As used herein, the term "computer-readable medium" may describe any form of suitable memory or memory device for retaining information in any form, including various kinds of storage devices (e.g., magnetic, optical, static, etc.). Indeed, various embodiments of the disclosure may be implemented in a wide variety of suitable forms.

<FIG> depicts a diagram of a system for augmenting ADAS features of a vehicle, in accordance with the claimed invention. In brief overview, system <NUM> includes an ADAS image SoC <NUM>. The ADAS image SoC <NUM> is associated with one or more image sensors <NUM>. The image sensors <NUM> capture images in and/or around the vehicle <NUM> and transmit the images to video/image processing module <NUM> of the ADAS image SoC <NUM>. The ADAS image SoC may also include a pattern recognizer <NUM>, pattern classifier <NUM>, an optical character recognition (OCR) module <NUM>, and/or a geometry analysis module <NUM>. Further, ADAS image SoC <NUM> may include a lane departure warning module <NUM>, a forward collision warning module <NUM>, and/or a traffic sign recognition module <NUM>. The ADAS image SoC <NUM> is in communication with an IVI system <NUM> of the vehicle <NUM>. The ADAS image SoC <NUM> may communicate with the IVI system <NUM> over a connection <NUM>, which may be a wired or wireless channel or mode. In some embodiments, connection <NUM> may be one of any wired connection as described herein. In some embodiments, connection <NUM> may be one of any wireless connection as described herein. Connection <NUM> may be a wireless connection. For example, connection <NUM> may be a radio link or an optical link. A radio link may include a Bluetooth connection, a Near Field Communication connection, a cellular data connection, a point-to-point connection, or a Wi-Fi connection. An optical link may include a fiber optic connection or an infrared data connection. The IVI system <NUM> may include a vision analysis module <NUM> and an ADAS enhancing module <NUM>. Further, the IVI system <NUM> may include a navigation engine <NUM> and/or a context engine <NUM>. The ADAS image SoC <NUM>, IVI system <NUM>, and ADAS controller <NUM> may communicate with each other via a bus, such as <NUM>, using at least one communication protocol, such as CAN BUS protocol.

Still referring to <FIG>, in greater detail, one or more image sensors <NUM>, such as cameras capture video in and/or around the vehicle <NUM>. In some embodiments, the image sensors <NUM> can capture continuous video. The video/image processing module <NUM> receives the images from the image sensors <NUM> and process single frames or a sample of frames. The video/image processing module <NUM> communicates with one or more of a pattern recognizer <NUM>, a pattern classifier <NUM>, OCR module <NUM>, and a geometry analysis module <NUM>.

The pattern recognizer <NUM> may perform a number of functions to process received images and identify patterns in an image or a series of images. A pattern classifier <NUM> may perform a number of functions to determine a kind of pattern recognized by the pattern recognizer <NUM>. An OCR module <NUM> may perform functions to convert an image to identify text. A geometry analysis module <NUM> may perform functions to process images using geometric algorithms and techniques. The modules may be in communication with the video / image processing module <NUM> and/or the various ADAS feature modules <NUM>, <NUM>, <NUM> of the ADAS image SoC <NUM>.

The ADAS image SoC <NUM> may include a lane departure warning (LDW) module <NUM>. The LOW module <NUM> may instruct the video/image processing module <NUM> to determine a distance of the wheel from the lane. This may be done by the video/image processing module <NUM> communicating with the pattern recognizer <NUM> and pattern classifier <NUM> to identify and determine lane markers in an image. The video/image processing module <NUM> may further communicate with the geometry analysis module <NUM> to process image geometry and vehicle geometry using information that may be embedded or otherwise associated with an image, such as camera optic information. The LOW module <NUM> may further receive information from the computing system <NUM> indicating the use of blinkers of the vehicle <NUM>. Based upon the processed image data and data received from computing system <NUM>, LOW module <NUM> may determine to generate a warning alarm to alert the driver that the vehicle is close to the edge of a lane.

The ADAS image SoC <NUM> may include a forward collision warning (FCW) module <NUM>. The FCW module <NUM> may instruct the video/image processing module <NUM> to detect objects near a vehicle <NUM>. For example, FCW module <NUM> may instruct the video/image process module <NUM> to detect objects in successive image samples. If there is a change in size of the identified objects relative to the speed of the vehicle, FCW module <NUM> may determine a probability of collision with the object. The speed of the vehicle <NUM> may be received from the computing system <NUM> of the vehicle <NUM>. The video/image processing module <NUM> may detect an object and track the relative size of the object using the geometry analysis module <NUM>. The FCW module <NUM> may determine to generate a warning alarm to alert the driver if the probability of collision with the identified object exceeds a predetermined threshold.

The ADAS image SoC <NUM> may include a traffic sign recognition (TSR) module <NUM>. The TSR module <NUM> may instruct the video/image processing module <NUM> to detect images. The video/image processing module <NUM> may process images received from the image sensors <NUM> using the OCR <NUM> module to identify text in the image and the pattern classifier <NUM> to identify a type of sign based upon the shape of the object. In some embodiments, the TSR module <NUM> may determine to alert the driver of the traffic sign.

In some embodiments, ADAS features may reside in the IVI system <NUM>. For example, ADAS features may reside on the ADAS enhancing module <NUM> of the IVI system <NUM>. ADAS features residing in or associated with the IVI system <NUM> may be non-critical safety features that would not require strict real-time performance. By leveraging the IVI system <NUM> for ADAS features of a vehicle <NUM>, the ADAS features residing in the computing system <NUM> of the vehicle <NUM> may increase the capability of the computing system <NUM> for critical safety features, such as lane departures or forward collision detection.

The vision analysis module <NUM> includes an image processing module <NUM>, a scene analysis feature extraction module <NUM>, a pattern recognition module <NUM>, and/or a heuristic analysis module <NUM>. The vision analysis module <NUM> may include other modules, which may not be shown, to facilitate image processing.

The image processing module <NUM> may perform image processing functionality, such as zooming, cropping, projection of images, and/or complex signal processing algorithms (e.g., linear filtering, pixelation, hidden Markov modeling). The scene analysis feature extraction module <NUM> processes received images to identify one or more objects in the image. A pattern recognition module <NUM> may perform image process images to identify patterns in the image or patterns in a series of images.

The heuristic analysis module <NUM> performs a number of functions to facilitate heuristic analysis of received images. Heuristic analysis may be a prior knowledge based analysis that determines or identifies risks based upon various decision rules or weighing methods. In some embodiments, the heuristic analysis module <NUM> may use machine-learning techniques. For example, thousands of previously classified test images may be processed in order to build a database of images to allow efficient analysis and categorization of individual items for future use.

Vision analysis module <NUM> may receive instructions from an ADAS enhancing module <NUM> regarding the processing of received images. The ADAS enhancing module <NUM> includes one or more of the following modules: wrong direction heading detection module <NUM>, rain detection and wiper control module <NUM>, and headlight control module <NUM>.

The vision analysis module <NUM> and/or ADAS enhancing module <NUM> may receive information from another source, such as a navigation engine <NUM>, a context engine <NUM>, a remote server <NUM>, an electronic device <NUM>, a cloud service, and/or any other type of data repository. The vision analysis module <NUM> may process received images based upon, at least in part, information received from one or more sources. The ADAS enhancing module <NUM> may determine an action based upon, at least in part, data received from the one or more sources, such as the navigation engine <NUM>, and the processed images received from the vision analysis module <NUM>.

ADAS enhancing module <NUM>, includes ADAS feature modules such as wrong direction heading detection (WDHD) module <NUM>, rain detection and wiper control (RDWC) module <NUM>, and/or headlight control (HLC) module <NUM>.

A WDHD module <NUM> may perform a number of functions to detect if the vehicle <NUM> is on the incorrect side of the road. For example, WDHD module <NUM> may receive or obtain information from a GPS device to determine the current driving direction in the current location of the vehicle <NUM>. WDHD module <NUM> may instruct vision analysis module <NUM> to determine whether the vehicle <NUM> is on the correct side of the road. Vision analysis module <NUM> processes received images based upon, at least in part, a pattern recognition module <NUM>, heuristic analysis module <NUM>, scene analysis feature extraction module <NUM>, and an OCR module (not shown) of the vision analysis module <NUM> to identify road signs, identify placement of road signs, detect lane markings, detect road dividers, among other factors. The WDHD module <NUM> may receive the processed images and related information as well as information from other sources, such as the GPS device and navigation engine <NUM>, to determine if the vehicle <NUM> is on the wrong side of the road. The WDHD module <NUM> may determine that the vehicle is on the incorrect side of the road and may determine an action based upon the determination. For example, the WDHD module <NUM> may determine the action to be generating an audible warning to the driver of the vehicle <NUM>. WDHD module <NUM> may then generate a message including the action and transmit the message to the ADAS controller <NUM> over the CAN bus <NUM>. The ADAS controller <NUM> may then communicate with the computing system <NUM> of the vehicle <NUM> and the computing system <NUM> may identify the appropriate component to enact the action identified by the WDHD module <NUM>.

Similarly, a RDWC module <NUM> may perform a number of functions to detect precipitation and responsive to identifying precipitation, may generate a message to transmit to the ADAS controller <NUM> to adjust the speed of the windshield wipers of the vehicle <NUM>. RDWC module <NUM> may receive images processed by one or more components of the vision analysis module <NUM> as well as information from other sources, such as a central server <NUM>, to determine an action.

As another example, HLC module <NUM> may perform a number of functions to control headlights in different environments and conditions. For example, HLC module <NUM> may direct the vehicle <NUM> to turn on/off headlights, brighten/dim headlights responsive to information received from a source, such as a navigation engine <NUM> and processed images received from the vision analysis module <NUM>. For example, images may be processed and HLC module <NUM> may use the processed images and information received for other sources, such as the navigation engine <NUM>, to distinguish between <NUM>-lane roads and multi-lane roads, presence of a road divider, identify approaching vehicles, determine presence of a tunnel. HLC module <NUM> may generate a message, including an action, to the ADAS controller <NUM> to control the function of the headlights responsive to the determination.

Further, the ADAS image SoC <NUM>, IVI system <NUM>, and ADAS controller <NUM> may communicate with each other via a communication protocol, such as a CAN bus <NUM>. For example, for safety-critical ADAS features, such as lane departure warnings, LOW module <NUM> may determine or identify an action based upon images processed by the video/image processing module <NUM>. The message, which may include the determined action, may be transmitted to the ADAS controller <NUM> via the bus <NUM>. The ADAS controller <NUM> may then communicate with the computing system <NUM> of the vehicle to enact the action identified by the LOW module <NUM>, such as emitting a warning signal and verbal message to the driver of the vehicle through a speaker of the vehicle <NUM>.

Similarly, for non-critical ADAS features, such as rain detection, RDWC module <NUM> may determine an action based upon images processed by the vision analysis module <NUM>. For example, the pattern recognition module <NUM> and scene analysis extraction module <NUM>, may process images received from the ADAS image SoC <NUM> to identify precipitation and its impact in visibility. The RDWC module <NUM> may determine an action based upon the processed images and information that may have been received from other sources, such as the navigation engine <NUM>. For example, the RDWC module <NUM> may determine that the windshield wipers should be turned on and the appropriate wiper speed. The message may then be transmitted by the ADAS enhancing module <NUM> to the ADAS controller <NUM> over the bus <NUM>. The ADAS controller <NUM> may then communicate with the computing system <NUM> of the vehicle to enact the action identified by the RDWC module <NUM>.

<FIG> illustrates an example flow diagram for augmenting ADAS features of a vehicle with image processing in on-board vehicle platform, according to one embodiment. In brief overview, the IVI system <NUM> receives <NUM> data captured by vehicle sensors <NUM>. The IVI system <NUM> processes <NUM> the received data. The IVI system <NUM> may analyze <NUM> the received data. The IVI system <NUM> may transmit <NUM> the analyzed data.

Still referring to <FIG>, in greater detail, the IVI system <NUM> receives <NUM> data captured by vehicle sensors <NUM>. For example, an ADAS image SoC <NUM> communicates with one or more image sensors 106a, 106b, 106c, to capture continuous video or multiple frames of images. Singles frames or a sample of frames are transferred from the ADAS image SoC <NUM> to the IVI <NUM>. The images may include additional data, such as a timestamp of when the image was captured or optical data of the image sensor <NUM>. Optical data may include data such as focal length of the image sensors <NUM> and resolution of the image sensor <NUM>. If the lens parameters of the image sensor <NUM> are fixed across all images, then the optical parameters may not need to be sent with every image.

The images from the ADAS image SoC <NUM> may be transmitted to the IVI system <NUM> at a low frequency. For example, the images may be transmitted from the ADAS image SoC <NUM> to the IVI system <NUM> at a rate of about <NUM> image/second. In some embodiments, images from the image sensors <NUM> may be transmitted in response to a triggering event identified by the ADAS image SoC <NUM>, such as a change in weather (e.g., if it begins raining). Images may be transmitted over an interface <NUM> between the ADAS image SoC <NUM> to the IVI system <NUM>. For example, the interface <NUM> may be a USB interface or a Media Oriented Systems Transport (MOST) bus.

In the claimed embodiments, the IVI system <NUM> processes <NUM> the received data. Processing the received images may be based upon, at least in part, data from an Internet connection, a database, a cloud service, a navigation system <NUM>, or a context engine <NUM>. In some embodiments, basic processing of the images may be performed by one or more components of the vision analysis module <NUM>. Components of the vision analysis module <NUM> include image processing <NUM>, scene analysis and feature extraction <NUM>, pattern recognition <NUM>, and/or heuristic analysis <NUM>.

In the claimed embodiments, the vision analysis module <NUM> processes <NUM> the images received from the ADAS image SoC <NUM>. As previously discussed, vision analysis module <NUM> may include an image processing component <NUM> which may provide image processing functionality, such as zooming, cropping, image projection, linear filtering, pixelation, and/or hidden Markov model (HMM). The images received from the ADAS image SoC <NUM> are processed by scene analysis feature extraction <NUM>, pattern recognition module <NUM>, and/or a heuristic analysis module <NUM>.

The ADAS enhancing module <NUM> determines an action based upon, at least in part, the processed data. For example, the ADAS enhancing module <NUM> may receive processed data from the vision analysis module <NUM>. Further, the ADAS enhancing module <NUM> may receive information from other sources, such as the navigation engine <NUM>, context engine <NUM>, a database, remote server <NUM>, electronic device <NUM>, a cloud service, and/or the Internet.

In some embodiments, data from the navigation engine <NUM> and/or context engine <NUM> may be used to enhance the functionality of the ADAS. For example, upcoming road curvature information from the navigation engine <NUM> may be used for head light control. A context engine <NUM> provides vehicle-to-vehicle information from other cars that may provide insight into weather conditions and may be used for wiper control or head light control.

The ADAS enhancing module <NUM> determines an action based upon the data received from the vision analysis module <NUM> and information received from other sources. The following examples are for illustrative purposes, and should not be construed as limitations of this disclosure. The examples illustrate scenarios in which images captured by an ADAS image sensors <NUM> (e.g., external facing camera of a vehicle <NUM>), may be used by the IVI system <NUM> to augment the vehicle ADAS offering.

In the claimed embodiments, a WDHD module <NUM> of the ADAS enhancing module <NUM> is used to detect if the vehicle <NUM> is heading in the wrong direction on a road as well as whether the vehicle is on the correct side of the road. A location of the vehicle <NUM> may be identified by a GPS device of the vehicle <NUM>. The location of the vehicle may be transmitted to the WRHD module <NUM>. Further, the navigation engine <NUM> may transmit additional information to the WRHD module <NUM> of the ADAS enhancing module <NUM>. The WRHD module <NUM> may determine whether the vehicle <NUM> is on a one-way road or two-way road and whether the vehicle <NUM> should be driven on the right-side or the left-side of the road. The images received from the ADAS image SoC <NUM> may then be analyzed and/or processed by the vision analysis module <NUM> to detect lanes, road dividers, pavements, and traffic signs which may determine whether the vehicle <NUM> is currently being driven on the correct side of the road or if the vehicle <NUM> is heading in the wrong direction on a one-way street. For example, the scene analysis feature extraction <NUM> module processes received images to identify the location of sidewalks and location of traffic signs in an image. The data indicating the identified location of sidewalks and traffic signs by the scene analysis feature extraction module <NUM> may be transmitted to the WRHD <NUM> module for further analysis. For example, the WRHD <NUM> may determine the vehicle <NUM> is driving on the wrong side of the road and determine <NUM> an action based upon, at least part, processed images and information from the navigation engine <NUM>.

In another claimed embodiment, a RDWC module <NUM> receives processed images from the vision analysis module <NUM>. The RDWC module <NUM> may apply further processing using techniques and algorithms that may be applied to images from the ADAS image sensors <NUM> to detect different types of precipitation (e.g., rain, snow, sleet, hail) and its severity. RDWC module <NUM> may determine <NUM> an action based upon, at least in part, the processed images and information that may be retrieved locally or retrieved from a source through a network connection. In one example, the action may be to adjust the wipers of the vehicle <NUM> based upon the detection of precipitation.

In another claimed embodiment, a HLC module <NUM> may detect the lighting condition surrounding the vehicle <NUM>. The HLC module <NUM> may communicate with the vision analysis module <NUM> to request specific processing of the images received from the ADAS image SoC. The HLC module <NUM> may request processing of images by the vision analysis module <NUM> captured by a particular image sensor <NUM> (e.g., a camera with a wider field of vision to better assess lighting conditions). The vision analysis module <NUM> may identify an on-coming vehicle by detecting the approaching vehicle's headlight while ensuring that road-side lights are not mistaken for vehicles through scene analysis and recognizing bright spots and estimating their distance from the images. Responsive to the scene analysis feature extraction module <NUM>, HLC module <NUM> may determine an action based upon the processed images. For example, the action determined by the HLC module <NUM> may be to dim head lights in the case of an oncoming vehicle on a multi-way road without a divider. In some embodiments, the action determined by the HLC module <NUM> may be to turn ON/OFF headlights based upon the level of ambient light identified by the vision analysis module <NUM>.

The IVI system <NUM> transmits <NUM> a message to an ADAS controller in response to determining or identifying the action. For example, the ADAS enhancing module <NUM> generates a message, based upon, at least in part, the determined action and transmit <NUM> the message to the ADAS controller. The ADAS controller receives and directs the message to the appropriate component of the computing system <NUM> to fulfill enact the determined action. For example, in the example in which the HLC module <NUM> identifies an oncoming vehicle and determines to dim the headlights, the ADAS enhancing module <NUM> may generate a message and transmit the message over the CAN bus to the ADAS controller <NUM> for the required action. Further, the ADAS controller <NUM> may transmit the message over the USB interface or MOST bus to the computing system <NUM> of the vehicle <NUM>. A component of the computing system <NUM> that controls the headlights of the vehicle <NUM> may receive the message identifying the action and enact the action.

Certain aspects of the present invention are described above with reference to block and flow diagrams systems and methods, wherein the present invention is defined by the appended claims.

It will be understood that one or more blocks of the block diagrams and flow diagrams, and combinations of blocks in the block diagrams and the flow diagrams, respectively, can be implemented by computer-executable program instructions. Likewise, some blocks of the block diagrams and flow diagrams may not necessarily need to be performed in the order presented, or may not necessarily need to be performed at all, according to some embodiments.

Claim 1:
A computer-implemented method for a system (<NUM>) in a vehicle, the system (<NUM>) comprising:
a vehicle computing system (<NUM>) configured to control the vehicle's standard devices or components, the vehicle computing system comprising an advanced driver assistance system, ADAS, image System on Chip, SoC, (<NUM>);
an in-vehicle infotainment, IVI, system (<NUM>) configured to provide entertainment and informational features for the vehicle, the IVI system comprising a processor (<NUM>), a memory (<NUM>), communication devices (<NUM>) and a transceiver (<NUM>);
one or more image sensors (<NUM>); and
an ADAS controller configured to communication with the computing system (<NUM>), ADAS image SoC (<NUM>) and IVI system (<NUM>);
wherein devices and/or program modules of the computing system (<NUM>) are configured to shut down when the vehicle is powered down while the processor (<NUM>), the memory (<NUM>), the communication devices (<NUM>), and the transceiver (<NUM>), of the IVI system (<NUM>) remain awake and provide certain functionality,
the method comprising:
receiving, by the processor (<NUM>) of the IVI system (<NUM>), from the ADAS image SoC (<NUM>), image data including images captured by the one or more image sensors (<NUM>);
processing, by the processor (<NUM>) of the IVI system (<NUM>), the image data which includes at least one of scene analysis and feature extraction, pattern recognition, and heuristic analysis;
determining, by the processor (<NUM>) of the IVI system (<NUM>), an action based upon, at least in part, the processed image data, the action related to one of wrong direction heading detection, rain detection and wiper control, and head light control; and
transmitting, by the processor (<NUM>) of the IVI system (<NUM>), a message to the ADAS controller in response to the determination; and
transmitting, by the ADAS controller, the message to an appropriate component of the computing system (<NUM>) to enact the determined action.