Automatic maintenance of front and/or rear windshield visibility

An apparatus comprising a camera sensor and a processor. The camera sensor may be configured to generate a video signal based on a targeted view in a vehicle. The processor may be configured to receive one or more status signals from one or more sensors. The processor may be configured to detect a type of obstruction of a window of the vehicle visible in the video signal based on (i) a classification of information in the video signal and (ii) one or more of the status signals.

FIELD OF THE INVENTION

The present invention relates to video capture devices generally and, more particularly, to a video capture device used to control automatic maintenance of front and/or rear windshields to improve visibility.

BACKGROUND OF THE INVENTION

One of the issues that drivers face when driving in various weather conditions is front windshield visibility. Poor front windshield visibility can be caused by dirt, by windshield glass freezing or by condensation on the windshield glass, or a “foggy windshield”. Drivers will usually handle poor visibility issues by either using wipers to clean the glass from the outside (in the case of dirt or glass freezing) or by using the temperature controls of the vehicle to heat the windshield above the dew point to clean condensation from the windshield. Clearing a “foggy windshield” is repeated as the glass becomes “foggy” again during driving, reducing visibility and distracting the driver from operating the vehicle. Reduced visibility and distracted driving pose a safety risk. Some conventional designs to improve visibility use dedicated humidity sensors to detect condensation. Such designs are expensive and have not achieved wide adoption.

It would be desirable to implement an automatic maintenance of front/rear windshield visibility that uses information from a video capture device that is cost effective.

SUMMARY OF THE INVENTION

The present invention concerns an apparatus comprising a camera sensor and a processor. The camera sensor may be configured to generate a video signal based on a targeted view in a vehicle. The processor may be configured to receive one or more status signals from one or more sensors. The processor may be configured to detect a type of obstruction of a window of the vehicle visible in the video signal based on (i) a classification of information in the video signal and (ii) one or more of the status signals.

The objects, features and advantages of the present invention include providing a video capture device that may (i) control maintenance of a front/rear windshield, (ii) use a shared video processor to reduce implementation costs, (iii) analyze video to detect conditions and initiate corrective measures, (iv) integrate information from vehicle sensors to diagnose a cause of a visibility obstruction and/or (v) be easy to implement.

DETAILED DESCRIPTION OF EMBODIMENTS

Referring toFIG. 1, a block diagram of an apparatus100is shown in accordance with an embodiment of the present invention. The apparatus100may be a camera system. The camera system100may comprise a block (or circuit)102, a block (or circuit)104, a block (or circuit)106and/or a block (or circuit)108. The circuit102may implement a capture device. The circuit104may implement an interface. The circuit106may be configured as a processor. The circuit108may be configured as a memory. The memory108may be configured to store computer readable/executable instructions (or firmware). The instructions, when executed by the processor106, may perform a number of steps.

The apparatus100is shown connected to a block (or circuit)110and/or a block (or circuit)114. The circuit110may be an external communication device. The circuit114may be implemented as one or more sensors (e.g., a location module such as a GPS sensor, an orientation module such as a magnetometer, a temperature sensor, etc.). Generally, the sensors114may be input/output devices separate from the capture device102. In some embodiments, the communication device110and/or the sensors114may be implemented as part of the camera system100(e.g., internal components of the camera system100). In some embodiments, the communication device110and/or the sensors114may be components available to the camera system100(e.g., pre-installed components of a vehicle).

The memory108is shown comprising a block (or circuit)116and a block (or circuit)118. The circuit116may be configured as a lookup table. The circuit118may be a data storage portion of the memory108. The memory108may comprise other portions (e.g., instructions, free memory, application-specific storage, shared memory, etc.). The type of data stored in the memory108may be varied according to the design criteria of a particular implementation.

The apparatus100is shown receiving input from a block (or circuit)112. The block112may be a lens (e.g., a camera lens). In some embodiments, the lens112may be implemented as part of the apparatus100. The components implemented in the apparatus100may be varied according to the design criteria of a particular implementation. In some embodiments, the apparatus100may be implemented as a drop-in solution (e.g., installed as one component).

The capture device102may present a signal (e.g., VIDEO) to the processor106. The interface104may present a signal (e.g., STATUS) to the processor106. The interface104is shown receiving data from the sensors114. The processor106may be configured to receive the signal VIDEO, the signal STATUS and/or other signals. The signal STATUS may present status information received from the sensors114. The processor106may be configured to generate a signal (e.g., CONTROL). The inputs, outputs and/or arrangement of the components of the camera system100may be varied according to the design criteria of a particular implementation.

The apparatus100may be implemented as a regular digital camera and/or a depth-sensing camera. The sensors114may comprise a GPS, a magnetometer, a temperature sensor, a humidity sensor, etc. The sensors114may be implemented on-board the camera system100and/or connected externally (e.g., via the interface104). The processor106may analyze the captured video content (e.g., the signal VIDEO) in real time to detect objects and/or obstructions (to be described in more detail in association withFIGS. 3-13).

Referring toFIG. 2, a block diagram of the apparatus100′ is shown in accordance with an embodiment of the present invention. The camera system100′ may comprise the capture device102′, the interface104, the processor106, the memory108, the communication device110, the lens112and/or the sensors114. The camera system100′ may be a distributed system (e.g., each component may be implemented separately throughout an installation location such as a vehicle). The capture device102′ may comprise a block (or circuit)120and/or a block (or circuit)122. The circuit120may be a camera sensor (e.g., a camera sensor separate from the sensors114). The circuit122may be a processor (e.g., a processor separate from the processor106). The capture device102′ may implement a separate internal memory (e.g., a memory separate from the memory108such as a frame buffer).

Referring toFIG. 3, a diagram illustrating obstructions and corrective measures in a frame150. A vehicle50is shown having a windshield160. The windshield160is in the video frame150. The frame150is shown as a targeted view from the vehicle50. The frame150shows obstructions156a-156b. The obstructions156a-156bmay be examples of various types of obstructions (e.g., obstructions156a-156n). Corrective measures152a-152nare shown. The number and/or types of the obstructions156a-156nand/or the corrective measures152a-152nmay be varied according to the design criteria of a particular implementation.

The obstructions156a-156nmay reduce a visibility through a window of the vehicle50(e.g., through the windshield160). An amount of the reduction in visibility may be related to the type of obstruction. In one example, the obstruction156amay be ice (e.g., frost on the windshield160). In another example, the obstruction156amay be fog (e.g., fog outside of the vehicle50). In yet another example, the obstruction156amay be dirt (e.g., mud splashed on the windshield). In still another example, the obstruction156bmay be water (e.g., washer fluid). One or more of the corrective measures152a-152nmay be applied to the windshield160to remove the obstruction156aand/or reduce visibility loss caused by the obstruction156a.

The corrective measures152a-152nmay be automatic responses by the vehicle50to counteract the obstructions156a-156n. The corrective measure152ais shown as washer fluid. For example, washer fluid (e.g., anti-freeze) may be used to clean dirt and/or ice from the windshield160. The corrective measures152b-152care shown as windshield wipers. For example, the wipers152b-152cmay be used to clean dirt, ice and/or rain from the windshield160. The corrective measures152d-152nare shown as being air vents (e.g., air vents for the heating/cooling system of the vehicle50). The air vents152d-152nmay be opened and/or aimed to release cold/hot air for the windshield160. For example, the air vents152d-152nmay be used to eliminate frost and/or condensation from the windshield160.

In some embodiments, the corrective measures152a-152nmay also be one of the obstructions156a-156n. For example, the washer fluid152a/156bmay be the corrective measure152ain response to mud, but the washer fluid152a/156bmay also be the obstruction156bthat causes a reduction in visibility. The wipers152b-152cmay need to be activated to remove the washer fluid152a/156b.

The capture device102may be configured to capture video image data (e.g., from the lens112). In some embodiments, the capture device102may be a video capturing device such as a camera. In some embodiments, the capture device102may be a component of a camera (e.g., a camera pre-installed at a fixed location such as a security camera). The capture device102may capture data received through the lens112to generate a bitstream (e.g., generate video frames). For example, the capture device102may receive light from the lens112. The lens112may be directed, panned, zoomed and/or rotated to provide a targeted view of the vehicle50(e.g., a field of view).

The capture device102may transform the received light into digital data (e.g., a bitstream). In some embodiments, the capture device102may perform an analog to digital conversion. For example, the capture device102may perform a photoelectric conversion of the light received by the lens112. The capture device102may transform the bitstream into video data, a video file and/or video frames (e.g., perform encoding). For example, the video data may be a digital video signal. The digital video signal may comprise video frames (e.g., sequential digital images).

The video data of the targeted view of the vehicle50may be represented as the signal/bitstream/data VIDEO (e.g., a digital video signal). The capture device102may present the signal VIDEO to the processor106. The signal VIDEO may represent the video frames/video data (e.g., the video frame150). The signal VIDEO may be a video stream captured by the capture device102. In some embodiments, the capture device102may be implemented in the camera. In some embodiments, the capture device102may be configured to add to existing functionality of the camera.

In some embodiments, the capture device102may be pre-installed at a pre-determined location and the camera system100may connect to the capture device102. In other embodiments, the capture device102may be part of the camera system100. The capture device102may be configured for driver monitoring, security monitoring, passenger monitoring, for insurance purposes, etc. For example, the capture device102may be implemented to detect break-ins and/or vandalism. In another example, the capture device102may detect accidents to provide evidence for insurance claims.

The capture device102may be configured to detect faces in a region of a video frame. In some embodiments, the capture device102may be configured to recognize faces through facial recognition (e.g., based on faces stored in the memory108). In some embodiments, the capture device102may be configured to detect objects and classify the objects as a particular type of obstruction. The camera system100may be configured to leverage pre-existing functionality of the pre-installed capture device102. The implementation of the capture device102may be varied according to the design criteria of a particular implementation.

In some embodiments, the capture device102′ may implement the camera sensor120and/or the processor122. The camera sensor120may receive light from the lens112and transform the light into digital data (e.g., the bitstream). For example, the camera sensor120may perform a photoelectric conversion of the light from the lens112. The processor122may transform the bitstream into a human-legible content (e.g., video data). For example, the processor122may receive pure (e.g., raw) data from the camera sensor120and generate (e.g., encode) video data based on the raw data (e.g., the bitstream). The capture device102′ may have a memory to store the raw data and/or the processed bitstream. For example, the capture device102′ may implement a frame memory and/or buffer to store (e.g., provide temporary storage and/or cache) one or more of the video frames (e.g., the digital video signal). The processor122may perform analysis on the video frames stored in the memory/buffer of the capture device102′.

In some embodiments the capture device102′ may be configured to determine a location of the obstructions156a-156n. For example, the processor122may analyze the captured bitstream (e.g., using machine vision processing), determine a location of the detected obstructions156a-156nand present the signal VIDEO (e.g., comprising information about the location of the detected obstructions156a-156n) to the processor106. The processor122may be configured to determine the location of the detected obstructions156a-156n(e.g., less analysis is performed by the processor106). In another example, the processor122may generate the signal VIDEO comprising video frames and the processor106may analyze the video frames to determine the location of the detected obstructions156a-156n(e.g., more analysis is performed by the processor106). The analysis performed by the processor122and/or the processor106may be varied according to the design criteria of a particular implementation.

In some embodiments, the processor122may be implemented as a local processor for the camera system100and the processor106may be implemented as an external processor (e.g., a processor on a device such as a server on a server farm). The processor122may be configured to combine the signal VIDEO and the signal STATUS for storage in the memory108(e.g., embed the status information, objects/obstructions156a-156nand/or metadata associated with the video frames in the video file as a text track, control channel, RTP stream, etc.). The camera system100′ may be configured to transmit the signal VIDEO with embedded status information to an external device (e.g., a device on an external network). The external device may have an external version of the processor106configured to perform the detection of the obstructions156a-156nand/or the determination of the corrective measures for the detected obstructions156a-156n.

The interface104may receive data from one or more of the sensors114. The signal STATUS may be generated in response to the data received from the sensors114at a time of generation of the signal VIDEO. In some embodiments, the interface104may receive data from a location module. In some embodiments, the interface104may receive data from an orientation module. In some embodiments, the interface104may receive data from a temperature module. In some embodiments, the interface104may receive weather information scraped from an external source (e.g., a weather service and/or website). In some embodiments, the interface104may receive data from the processor106and/or the communication device110. The interface104may send data (e.g., instructions) from the processor106to connected devices via the communications device110. For example, the interface104may be bi-directional.

In the examples shown (e.g., inFIG. 1andFIG. 2), information from the sensors114(e.g., the location module, the orientation module, the temperature module, etc.) may be received by the interface104. In one example, where the camera system100is installed in a vehicle, the interface104may be implemented as an electronic bus (e.g., a controller area network (CAN) bus) and the sensors114may be part of the vehicle. In another example, the interface104may be implemented as an Ethernet interface. In yet another example, the interface104may be implemented as an electronic device (e.g., a chip) with a CAN bus controller. In some embodiments, the sensors114may connect directly to the processor106(e.g., the processor106may implement a CAN bus controller for compatibility, the processor106may implement a serial peripheral interface (SPI), the processor106may implement another interface, etc.). In some embodiments, the sensors114may connect to the memory108.

The processor106may be configured to execute computer readable code and/or process information. The processor106may be configured to receive input and/or present output to the memory108. The processor106may be configured to present and/or receive other signals (not shown). The number and/or types of inputs and/or outputs of the processor106may be varied according to the design criteria of a particular implementation.

In some embodiments, the processor106may receive the signal VIDEO from the capture device102and detect the objects156a-156nin the video frame. In some embodiments, the processor122may be configured to detect the obstructions156a-156nand the processor106may receive the location (or coordinates) of the detected obstructions156a-156nin the video frame from the capture device102′. In some embodiments, the processor106may be configured to analyze the video frame (e.g., the signal VIDEO). The processor106may be configured to detect a location and/or position of the detected obstructions156a-156nin the video frame.

The processor106may determine a type of the detected obstructions156a-156nbased on a classification. The classification may be based on information from the signal VIDEO (e.g., object detection) and/or information from the signal STATUS (e.g., environmental factors). For example, the color histogram and/or the high frequency component of the signal VIDEO may be compared to some known reference. In another example, temperature and/or humidity information may be provided by the signal STATUS. The processor106may rule out and/or increase a likelihood of certain types of obstructions. For example, the classification may comprise a confidence level for a particular hypothesis (or diagnosis) for the cause of visibility reduction.

A high confidence level for a particular type of obstruction may indicate that evidence (e.g., from the signal VIDEO and/or STATUS) is consistent with the particular type of obstruction. A low confidence level for a particular type of obstruction may indicate that evidence (e.g., from the signal VIDEO and/or STATUS) is inconsistent with the particular type of obstruction. Various checks may be performed to determine the confidence level. The corrective measures152a-152nmay be activated when a confidence level is above a pre-defined threshold. The implementation of the classification to determine the type of obstruction may be varied based on the design criteria of a particular implementation.

Based on the location and/or the classification of the detected obstructions156a-156nin the video frame (e.g., the signal VIDEO), the processor106may determine the appropriate response and/or corrective measure for the type of the obstructions156a-156n. The corrective measures152a-152nfor the detected obstructions156a-156nmay be based on the signal VIDEO and/or the signal STATUS. The processor106may generate the signal. CONTROL in response to the determined corrective measures152a-152nfor the detected obstructions156a-156n. The corrective measures152a-152nmay be an automatic response by the vehicle50.

The signal CONTROL may be implemented to provide an activation for the corrective measures152a-152nin response to the classification of the obstructions156a-156n. For example, the signal CONTROL may be sent to the interface104in order to activate the appropriate device to initiate the corrective measure (e.g., windshield wipers, conductive resistors, washer fluid, fog lights, etc.). Generally the signal CONTROL may correspond to the type of the classified obstructions156a-156n.

The utilization of the data stored in the signal CONTROL may be varied according to the design criteria of a particular implementation. In some embodiments, the signal CONTROL may be presented to the communication device110and the communication device110may pass the signal CONTROL to an external network and/or external storage. For example, if the obstruction156a-1.56nis a cracked or smashed windshield, the signal CONTROL may be sent to a roadside assistance service (e.g., a towing service, emergency services, etc.), an insurance provider and/or a mechanic.

The processor106and/or the processor122may be implemented as an application specific integrated circuit (e.g., ASIC) or a system-on-a-chip (e.g., SOC). The processor106and/or the processor122may be configured to determine a current size, shape and/or color of the obstructions156a-156n(e.g., to perform a classification). The processor106and/or the processor122may detect one or more of the detected objects156a-156nin each video frame. In some embodiments, the processor106and/or the processor122may receive video signals from multiple cameras and/or image sensors.

The processor106and/or the processor122may determine a number of pixels (e.g., a width, a height and/or a depth) comprising the detected obstructions156a-156nin the video frame. Based on the number of pixels of each of the detected objects156a-156nin the video frame, the processor106and/or the processor122may estimate a classification of the detected obstructions156a-156n. Whether the detection of the objects156a-156nis performed by the processor106and/or the processor122may be varied according to the design criteria of a particular implementation.

The memory108may store data. The memory108may be implemented as a cache, flash memory, DRAM memory, etc. The type and/or size of the memory108may be varied according to the design criteria of a particular implementation. The data stored in the memory108may correspond to the detected obstructions156a-156n, reference objects, a video file, status information (e.g., readings from the sensors114) and/or metadata information.

For example, the memory108(e.g., the lookup table116) may store a reference size (e.g., the number of pixels of an object of known size in a video frame at a known distance) of the objects156a-156n. In another example, the memory108may store a reference shape (e.g., an arrangement of pixels of the reference object in the video frame at a known distance). In yet another example, the memory108may store a reference color (e.g., a RGB value and/or a YCbCr value for each of the pixels of the reference object in the video frame) of the objects156a-156n. The reference size, shape and/or colors stored in the memory108may be used to compare the current size of the detected obstructions156a-156nin a current video frame. The comparison of the size of the detected obstructions156a-156nin the current video frame and the reference size, shape and/or color may be used to estimate a classification of the obstructions156a-156n.

The memory108may store the pre-determined location of the camera system100and/or a pre-determined field of view of the camera system100(e.g., when the camera system100is implemented as a fixed view camera). The memory108may store reference data for the obstructions156a-156n. For example, the memory108may store reference color histograms for various known types of obstructions. In another example, the memory108may store previously capture frames (e.g., a reference image from when a car was parked and turned off). The type of reference information stored by the memory108may be varied according to the design criteria of a particular implementation.

The memory108may store the lookup table116. The lookup table116stored in the memory108may comprise reference information. The lookup table may allow the signal VIDEO and/or the signal STATUS to be compared to and/or cross-referenced with some known set of data. Generally, the lookup table116may be implemented to index precalculated values to save computation time. For example, the lookup table116may store temperature values, dew point values and/or humidity values. The detected temperature and/or humidity values may be compared to values in the lookup table116to perform a classification of the obstructions156a-156nand/or activate one of the corrective measures152a-152n. In one example, data values for the lookup table116may be scraped (e.g., using the communication device110) from a weather service. In another example, data values for the lookup table116may be precalculated (e.g., during an idle time of the processor106).

The communication device110may send and/or receive data to/from the interface104. In some embodiments, when the camera system100is implemented as a vehicle camera, the communication device110may be the OBD of the vehicle. In some embodiments, the communication device110may be implemented as a satellite (e.g., a satellite connection to a proprietary system). In one example, the communication device110may be a hard-wired data port (e.g., a USB port, a mini-USE port, a USB-C connector, HDMI port, an Ethernet port, a DisplayPort interface, a Lightning port, etc.). In another example, the communication device110may be a wireless data interface (e.g., Wi-Fi, Bluetooth, ZigBee, cellular, etc.).

The lens112(e.g., a camera lens) may be directed to provide a targeted view of the vehicle50and/or the environment (e.g., a field of view from the camera sensor102and/or an external camera sensor). In one example, the lens112may be mounted on a dashboard of the vehicle50. In another example, the lens112may be wearable camera (e.g., a camera worn by a police officer, a camera worn by a race car driver, a camera worn by a first responder, a camera worn by a thrill-seeker, etc.). The lens112may be aimed to capture environmental data (e.g., light). The lens112may be configured to capture and/or focus the light for the capture device102. Generally, the sensor120is located behind the lens112. Based on the captured light from the lens112, the capture device102may generate a bitstream and/or video data.

The sensors114may be configured to determine a location and/or an orientation of the camera system100. The number and/or types of data used to determine the location and/or orientation of the camera system100may be varied according to the design criteria of a particular implementation. In one example, the location module may be used to determine an absolute location of the camera system100. In another example, the orientation module may be used to determine an orientation of the camera system100. Other types of sensors may be implemented. For example, a temperature module may be implemented to determine an inside and/or an outside temperature for the vehicle50. Sensors on the windshield may be used to determine the inside temperature of the glass of the windshield160. In another example, a humidity sensor may be implemented to determine a humidity level of the environment.

Data from the sensors114may be presented to the processor106as the signal STATUS. The number and/or types of the sensors114may be varied according to the design criteria of a particular implementation. The sensors114may be used by the camera system100to determine (e.g., confirm a likelihood) a type of the obstructions156a-156n(e.g., confirm classifications).

The sensors114(e.g., the location module, the orientation module and/or the other types of sensors) may be configured to determine an absolute location and/or an azimuth orientation of the camera system100. The absolute location and/or the azimuth orientation of the camera system100may be added to the detected relative location of the obstructions156a-156nto determine an absolute location (e.g., coordinates) of the obstructions156a-156n. The absolute location of the vehicle50and/or the absolute location of the obstructions156a-156nmay be used to determine the type of the obstruction156a-156n.

The signal STATUS may provide information for the camera system100(e.g., the status information). In one example, location information may be determined by the location module (e.g., to determine weather conditions for the current location of the vehicle50). For example, the location module may be implemented as a GPS sensor. Orientation information may be determined by the orientation module. For example, the orientation module may be implemented as a magnetometer, an accelerometer and/or a gyroscope. In yet another example, temperature information may be determined by the temperature module. For example, the temperature module may be implemented as a thermometer.

The types of sensors used to implement the location module, the orientation module, the temperature module and/or any other types of sensors may be varied according to the design criteria of a particular implementation. In some embodiments, the signal STATUS may provide details about the camera system100(e.g., camera specifications, camera identity, the field of view204, date, time, etc.).

Referring toFIG. 4, a diagram illustrating a frame150′ and a frame150″ is shown. The frame150′ may show details of a rear windshield170. The frame150″ may also show details of a side window180. The rear windshield170is shown having frost156i. The frost156ireduces visibility of the rear windshield170. The side window180is shown having frost156n. The frost156nreduces visibility of the side window180.

The camera system100may use information from the rear windshield170and/or the side window180to infer a classification of the type of obstruction on another window (e.g., the front windshield160). For example, the frost156idetected on the rear windshield170and/or the frost156ndetected on the side window180may increase the confidence level of a detection of frost on the front windshield160(e.g., all windows are likely to be obstructed when frost is the cause of the reduction in visibility). In another example, if the obstruction on the windshield160is caused by mud and/or dirt, the rear windshield170and/or the side window180may not have a similar obstruction.

The rear windshield170is shown having one of the corrective measures152a-152n(e.g.,152i). The corrective measure152imay be implemented as resistive conductors in or on the glass of the rear windshield170(e.g., defrost rails). Similar resistive conductors may be implemented on the front windshield160(or portions of the front windshield160, such as where the windshield wipers152b-152crest). Further examples of the corrective measures152a-152nmay be implemented on the other windows170and/or180. For example, the rear windshield170may be configured to have a washer fluid nozzle and/or a wiper blade. The types of corrective measures implemented for each of the windows of the vehicle50may be varied according to the design criteria of a particular implementation.

Referring toFIG. 5, a side view of the automobile50is shown. The sensor102is shown having an angle190aand an angle190b(e.g., a field of view) that points toward the frame150. The field of view from the angle190aand the angle190bmay provide a targeted view of the vehicle50. The sensor102is also shown having an angle192aand an angle192b(e.g., a field of view) that points toward the frame150′. The field of view from the angle192aand the angle192bmay provide a targeted view of the vehicle50. The sensor102may also point at the frame150″. The frame150, the frame150′ and the frame150″ show images of the various windshields and/or windows of the vehicle50.

The vehicle50may have an external camera sensor200. The external sensor200may provide a targeted view from the vehicle50(e.g., a front view camera providing a targeted view in front of the windshield160, a rear-view camera, a side-view camera, etc.). The external camera sensor200may be similar to the camera sensor102. For example, the external camera200may have a separate sensor from the camera system100and/or provide a second video signal. The external camera sensor200is shown having an angle202aand an angle202b(e.g., a field of view) that points away from the vehicle50. The field of view from the angle202a-202bmay provide a targeted view from the vehicle50. In some embodiments, the external camera sensor200may be directed towards the vehicle50. The implementation of the external camera sensor200may be varied according to the design criteria of a particular implementation.

The external camera sensor200may be used by the processor106to compare an obstruction detected by the processor106on one of the windows of the vehicle. For example, if the processor106detects an obstruction on the windshield160that may be caused by fog, the external camera sensor200may be used to confirm that the obstruction is fog (e.g., both the windshield160and the external camera sensor200detect the same obstruction). In another example, if the obstruction is caused by dirt, the windshield160may have an obstruction, but the external camera sensor200may not have the obstruction.

One of the corrective measures152a-152n(e.g.,152j) is shown on the vehicle50. The corrective measure152jis shown as activated (e.g., flashing) headlights (or high beams or fog lights). The corrective measure152jmay be used to in response to a detected obstruction (e.g., fog). In some embodiments, the corrective measure152jmay be used to determine (e.g., test a diagnosis or hypothesis) the type of obstruction156a-156n. For example, the headlights152jmay be activated in order for the camera system100to detect a response of the obstructions156a-156ndue to reflections caused by the headlights152j.

Referring toFIG. 6, a method (or process)300is shown. The method300may apply a corrective measure based on an obstruction type. The method300generally comprises a step (or state)302, a step (or state)304, a decision step (or state)306, a step (or state)308, a step (or state)310, a step (or state)312, and a step (or state)314. The steps302-314may be performed by the processor106in response to computer executable instructions stored in the memory108.

The step302starts the method300. The step304monitors the window160in the video frame150to determine reduced visibility. Next, the decision step306determines whether an obstruction156has been detected. If not, the method300moves back to the step304. If so, the method300moves to the step308. The step308classifies information in the video frame150. The step310receives status information from the sensors114. Next, the step312determines a type of the obstruction156. Next, the step314applies corrective measures based on the type of the obstruction156. The method300then moves back to the step304.

Referring toFIG. 7, a method (or process)350is shown. The method350may be an iterative application of corrective measures. The method350generally comprises a step (or state)352, a step (or state)354, a step (or state)356, a step (or state)358, a decision step (or state)360, a step (or state)362, and a step (or state)364. The steps352-364may be performed by the processor106in response to computer executable instructions stored in the memory108.

The step352starts the method350. Next, the step354determines whether an obstruction has been detected. Next, the step356determines the type of obstruction. Next, the step358activates one or more corrective measures based on the type of the obstruction156detected. Next, the decision step360determines whether visibility has improved. If so, the method350moves to the step364, which ends the method350. If visibility has not improved, the method350moves to the step362. The step362determines an alternate obstruction type. The method350then moves back to the step358.

Referring toFIG. 8, a method (or process)400is shown. The method400may be an example of an iterative determination of an obstruction type. The method400generally comprises a step (or state)402, a step (or state)404, a step (or state)406, a decision step (or state)408, a step (or state)410, a decision step (or state)412, a step (or state)414, a step (or state)416, a step (or state)418, a step (or state)420, a decision step (or state)422, a step (or state)424, a step (or state)426, a decision step (or state)428, a step (or state)430, a step (or state)432, and a step (or state)434. The steps402-434may be performed by the processor106in response to computer executable instructions stored in the memory108.

The state402may start the method400. The state404may detect one of the obstructions156a-156nin the window160(or the windows170,180, etc.). Next, the state406may check a color histogram of the obstructions156a-156n. Next, the decision state408determines if the colors are shifted from expected channels (e.g., the color histogram of the image may be compared to some reference such as an the external camera200, a view of the environment stored in the memory108(e.g., an image/video taken earlier when the car was parked, before the vehicle50was turned off, from the external camera200and/or an image/video from a similar time of day), a reference obstruction, etc.). If not, the method400moves to the state410.

The state410determines the obstruction may likely be ice (e.g., the confidence level associated with ice may be increased and/or the confidence level associated with water may be increased). Next, the method400may move to the decision state412. The decision state412may determine if there is enough time to defrost the windshield (e.g., using a slower corrective measure such as activating heat through the vents152d-152n). If so, the method400may move to the state414. If not, the method400may move to the state416. The state414may activate heat to de-ice the windshield (e.g., a primary defogger such as heat through the vents152d-152nand/or a secondary defogger such as the defrost rails152i) as the corrective measure. Next, the method400may move to the state434. The state416may activate the washer fluid152a, the wipers152b-152cand/or the defrost rails152ias the corrective measure. Next, the method400may move to the state434.

If the decision state408determines the colors are shifted from the expected channels, the method400may move to the state418(e.g., the confidence level for ice may be decreased). The state418may activate the headlights152jof the vehicle50. The state420may check the signal VIDEO for reflections (e.g., a response of the obstructions156a-156nin response to the corrective measure152j). Next, the method400may move to the decision state422.

If the decision state422determines the reflection indicates a large smear (e.g., the obstructions156a-156nare shown as a large smear), the method400may move to the state424. The state424may determine the obstruction may likely be fog (e.g., increase the confidence level associated with fog). In the state426, fog lights of the vehicle50may be activated as one of the corrective measures152a-152n. After the state426, the method400may move to the state434. If the decision state422determines that the reflection does not indicate a large smear, the method400may move to the decision state428(e.g., the confidence level for fog may be decreased).

If the decision state428determines the obstruction does not have gray or brown color tones, the method400may move to the state430(e.g., the confidence level associated with condensation may increase and/or the confidence level associated with mud/dirt may decrease). The state430may activate defoggers (e.g., using the air vents152d-152n) to change a temperature of the inside of the windshield160to above the dew point. Next, the method400may move to the state434. If the decision state428determines the obstruction does have gray or brown color tones, the method400may move to the state432(e.g., the confidence level associated with mud/dirt may increase and/or the confidence level for condensation may decrease). The state432may activate the washer fluid152aand/or the wipers152b-152cas the corrective measure. Next, the method400may move to the state434. The state434may end the method400.

The camera sensor102inside the vehicle50may be used to automate the process that a driver normally performs manually. The camera sensor102may be implemented as a camera sensor configured to see a targeted view of the windshield160from inside the vehicle50. The camera sensor102may be implemented as a dash camera, a driver assistance camera, an interior security camera, etc. Since the same process can be applied for the rear windshield170, or the side window180, a driver monitoring camera may be used.

The apparatus100may be used to detect poor visibility in the vehicle50. The apparatus100may be used to detect “fog” or other poor visibility conditions. The apparatus100may use the processor106to analyze a histogram and/or a high frequency component of the frame150(or150′ or150″) to detect the obstructions156a-156n. For example, a color histogram may be compared to a reference color histogram to see if the detected colors are shifted compared to reference colors (e.g., no shift in color may indicate ice, a shift to a whiter than usual color may indicate fog and/or condensation, etc.). In another example, the high frequency component may be used to detect details such as sharp edges to indicate whether or not a blur is detected (e.g., a blur caused by fog). In yet another example, gray and/or brown color tones may indicate dirt and/or mud rather than condensation. The apparatus100may be used with modern driver assistance cameras to incorporate computer vision algorithms that may be used to detect a variety of objects in the scene and/or poor visibility (e.g., obstructions that reduce and/or limit visibility from inside the vehicle50).

In order to start one or more countermeasures (e.g., the corrective measures152a-152n), the apparatus100may determine whether an obstruction is frozen glass, dirty glass, foggy glass outside fog, etc. (e.g., classify the obstruction152a-152n). Rain/snow may be detected in modern cars, which have automatic wiper control and/or through computer vision means. Detecting the type of obstruction152a-152nmay be done by implementing one or more procedures or checks. For example, if the temperature inside the vehicle50is above a likely dew point, the window may not be “foggy”.

A dew point may be calculated in response to sensor inputs (e.g., status information from the sensors114) such as air temperature and/or relative humidity measurements. For example, the sensors114may provide inputs (e.g., the status information) that may be approximated either through available humidity sensors and/or GPS location/weather data. The obstructions156a-156nmay be classified based on the status information. For example, if the temperature outside is sufficiently above freezing, the window160is not frozen.

A dew point may be determined based on a measure of atmospheric moisture. Generally, the dew point is a temperature that dew will form when the air temperature falls sufficiently. When temperatures are below freezing, the dew point may be a frost point (e.g., the point at which ice may form the obstruction on the windshield). In some embodiments, the camera system100may calculate the dew point and/or frost point (based on information received from the sensors114). The dew point and/or frost point may be stored in the lookup table116of the memory108. For example, the lookup table116may store dew point values based on air temperature and relative humidity values. The air temperature value and the relative humidity value may be cross-referenced to point to a precalculated dew point temperature. In some embodiments, the dew point and/or frost point may be received from an external service (e.g., a weather provider) using the communication device110.

Referring toFIG. 9, a method (or process)450is shown. The method450may be an example of classifying a type of obstruction based on a temperature check. The method450generally comprises a step (or state)452, a step (or state)454, a step (or state)456, a decision step (or state)458, a step (or state)460, a decision step (or state)462, a step (or state)464, a step (or state)466, a decision step (or state)468, a step (or state)470, a step (or state)472, and a step (or state)474. The steps452-474may be performed by the processor106in response to computer executable instructions stored in the memory108.

The state452may start the method450. In the state454, the camera system100may detect one or more of the obstructions156a-156n. In the state456, the processor106may read the status information (e.g., a temperature reading). Next, the method450may move to the decision state458.

If the decision state458determines the temperature of the inside of the windshield160is above the dew point, the method450may move to the state460. In the state460, the classification of the obstructions156a-156nmay be considered likely to not be fog (e.g., the confidence level associated with fog may be decreased). Next, the method450may move to the state466. If the decision state458determines the temperature of the inside of the windshield160is not above the dew point, the method450may move to the decision state462.

If the decision state462determines the temperature outside is above freezing, the method450may move to the state464. In the state464, the classification performed by the processor106may determine the obstructions156a-156nare not likely to be ice (e.g., the confidence level associated with ice may be decreased and/or the confidence level associated with fog may be increased). Next, the method450may move to the state466. If the decision state462determines the temperature outside is not above freezing, the method450may move to the state466(e.g., the confidence level associated with ice may be increased).

The state466may aggregate information about the obstructions156a-156n(e.g., the processor106may aggregate information from one or more checks performed as described inFIGS. 9-13). Next, the method450may move to the decision state468. If the decision state468determines the obstructions156a-156ncan be classified, the method450may move to the state470. The state470may perform one or more of the corrective measures152a-152nbased on the classification. Next, the method450may move to the state474. If the decision state468determines the obstructions156a-156ncannot be classified, the method450may move to the state472. The state472may perform other classification checks. Next, the method450may end at the state474.

The speed of deterioration of visibility (or a change in the obstruction) may also be used to calculate an appropriate corrective measure. For example, if the visibility declines quickly during driving, then the windshield160may not be frozen. Quick deterioration may indicate the obstruction is likely “foggy” glass and/or outside fog. Computer vision may also be used to calculate the corrective measures152a-152n.

The processor106may be used to assess the likelihood of each type of problem (e.g., obstruction) by analyzing the way the glass looks. Based on the assessment, the obstruction may be classified (e.g., a type of the obstruction152a-152nmay be determined). For example, a dirty window may likely have darker colors and/or a very non-uniform image. In another example, a strong fog may likely be brighter and/or more uniform.

Referring toFIG. 10, a method (or process)500is shown. The method500may be an example of classifying a type of obstruction based on a speed of visibility deterioration check. The method500generally comprises a step (or state)502, a step (or state)504, a step (or state)506, a decision step (or state)508, a step (or state)510, a step (or state)512, a decision step (or state)514, a step (or state)516, a step (or state)518, and a step (or state)520. The steps502-520may be performed by the processor106in response to computer executable instructions stored in the memory.

The state502may start the method500. In the state504, the camera system100may detect one or more of the obstructions156a-156n. The state506may monitor visibility changes of the obstructions156a-156nin the signal VIDEO. Next, the method500may move to the decision state508.

If the decision state508determines the visibility is deteriorating quickly, the method500may move to the state510. In the state510, the classification performed by the processor106may determine the obstructions156a-156nare not likely to be ice (e.g., the confidence level associated with ice may be decreased). Next, the method500may move to the state512. If the decision state508determines the visibility is not deteriorating quickly, the method500may move to the state512(e.g., the confidence level associated with ice may be increased).

The state512may aggregate information about the obstructions156a-156n(e.g., the processor106may aggregate information from one or more checks performed as described inFIGS. 9-13). Next, the method500may move to the decision state514. If the decision state514determines the obstructions156a-156ncan be classified, the method500may move to the state516. The state516may perform one or more of the corrective measures152a-152nbased on the classification. Next, the method500may move to the state520. If the decision state514determines the obstructions156a-156ncannot be classified, the method500may move to the state518. The state518may perform other classification checks. Next, the method500may end at the state520.

The condition on one or more windshields/windows may be used to determine a particular corrective measure to evoke. The camera system100in the vehicle50may be configured to see and use information from multiple windows (e.g., the windows160,170,180, etc.) to calculate multiple frames150. For example, if only the front windshield160does not provide good visibility, the cause of reduced visibility (e.g., the obstruction) may likely be dirt. The external camera200(such as rear view cameras, surround view cameras, etc.) may be used to compare. For example, if the visibility of the external camera200is much better, then a classification of fog and/or poor visibility may be unlikely.

Referring toFIG. 11, a method (or process)550is shown. The method550may be an example of classifying a type of obstruction based on an information check from other windows and cameras. The method550generally comprises a step (or state)552, a step (or state)554, a step (or state)556, a decision step (or state)558, a step (or state)560, a decision step (or state)562, a step (or state)564, a step (or state)566, a decision step (or state)568, a step (or state)570, a step (or state)572, and a step (or state)574. The steps552-574may be performed by the processor106in response to computer executable instructions stored in the memory108.

The state552may start the method550. In the state554, the camera system100may detect one or more of the obstructions156a-156n. The state556may check other cameras (e.g., the external camera200) and/or other video frames (e.g., the rear window video frame150′ and/or the side window video frame150″) for obstructions. Next, the method550may move to the decision state558.

If the decision state558determines only the front windshield160has the obstructions156a-156n, the method560may move to the state560. In the state560, the classification performed by the processor106may determine the obstructions156a-156nare likely to be dirt (e.g., the confidence level associated with dirt may be increased). Next, the method550may move to the state566. If the decision state558determines that the windshield160is not the only window that has obstructions, the method550may move to the decision state562(e.g., the confidence level associated with dirt may be decreased and/or the confidence level with other types of obstructions may be adjusted).

If the decision state562determines that the external camera200does not have reduced visibility, the method550may move to the state564. In the state564, the classification performed by the processor106may determine the obstructions156a-156nare not likely to be fog (e.g., the confidence level associated with fog may be decreased). Next, the method550may move to the state566. If the decision state562determines that the external camera does have reduced visibility, the method550may move to the state566(e.g., the confidence level associated with fog may be increased).

The state566may aggregate information about the obstructions156a-156n(e.g., the processor106may aggregate information from one or more checks performed as described inFIGS. 9-13). Next, the method550may move to the decision state568. If the decision state568determines the obstructions156a-156ncan be classified, the method550may move to the state570. The state570may perform one or more of the corrective measures152a-152nbased on the classification. Next, the method550may move to the state574. If the decision state568determines the obstructions156a-156ncannot be classified, the method550may move to the state572. The state572may perform other classification checks. Next, the method550may end at the state574.

Referring toFIG. 12, a method (or process)600is shown. The method600may be an example of classifying a type of obstruction based on a color and uniformity of the obstruction check. The method600generally comprises a step (or state)602, a step (or state)604, a step (or state)606, a decision step (or state)608, a step (or state)610, a decision step (or state)612, a step (or state)614, a step (or state)616, a decision step (or state)618, a step (or state)620, a step (or state)622, and a step (or state)624. The steps602-624may be performed by the processor106in response to computer executable instructions stored in the memory108.

The state602may start the method600. In the state604, the camera system100may detect one or more of the obstructions156a-156n. The state606may monitor color and/or uniformity of the obstructions156a-156nin the signal VIDEO. Next, the method600may move to the decision state608.

If the decision state608determines the colors of the obstructions156a-156nare dark and non-uniform, the method600may move to the state610. In the state610, the classification performed by the processor106may determine the obstructions156a-156nare likely to be dirt on the window160(e.g., the confidence level associated with dirt may be increased). Next, the method600may move to the state616. If the decision state608determines the colors of the obstructions156a-156nare not dark and non-uniform, the method600may move to the decision state612(e.g., the confidence level associated with dirt may be decreased).

If the decision state612determines the colors of the obstructions156a-156nare light and uniform, the method600may move to the state614. In the state614, the classification performed by the processor106may determine the obstructions156a-156nare likely to be a foggy window160(e.g., the confidence level associated with foggy windshield may be increased). Next, the method600may move to the state616. If the decision state612determines the colors of the obstructions156a-156nare not light and uniform, the method600may move to the state616(e.g., the confidence level associated with a foggy windshield may be decreased).

The state616may aggregate information about the obstructions156a-156n(e.g., the processor106may aggregate information from one or more checks performed as described inFIGS. 9-13). Next, the method600may move to the decision state618. If the decision state618determines the obstructions156a-156ncan be classified, the method600may move to the state620. The state620may perform one or more of the corrective measures152a-152nbased on the classification. Next, the method600may move to the state624. If the decision state618determines the obstructions156a-156ncannot be classified, the method600may move to the state622. The state622may perform other classification checks. Next, the method600may end at the state624.

A number of active detection measures may be used. The active detection measures may comprise activating one or more of the corrective measures152a-152nto help perform a classification of the type of the obstructions156a-156n. For example, activating and/or flashing headlights (or high beams) may be used as one of the active detection measures. If the obstruction is outside fog, the camera capture device102may be able to detect a reflection in response to activating and/or flashing the headlights152j. Wipers and/or washer fluid may also be used as an active detection measure. A notable change in visibility may be measured when dealing with dirt and/or frozen glass. Using A/C to heat the windshield above dew point may also be implemented as an active detection measure. If “foggy” glass is sensed, the visibility should rapidly improve with A/C.

Referring toFIG. 13, a method (or process)650is shown. The method650may be an example of classifying a type of obstruction based on responses to corrective measures. The method650generally comprises a step (or state)652, a step (or state)654, a step (or state)656, a decision step (or state)658, a decision step (or state)660, a step (or state)662, a step (or state)664, a decision step (or state)666, a step (or state)668, and a step (or state)670. The steps652-670may be performed by the processor106in response to computer executable instructions stored in the memory108.

The state652may start the method650. In the state654, the camera system100may detect one or more of the obstructions156a-156n. Next, the state656may perform a first classification test (or check, such as one of the checks described inFIGS. 9-12and/or other checks). Next, the method650may move to the decision state658.

If the decision state658determines the obstructions156a-156ncan be classified, the method650may move to the state668. If the decision state658determines the obstructions156a-156ncannot be classified, the method650may move to the decision state660. If the decision state660determines there are other classification tests, the method650may move to the state662. The state662may perform a next classification test (or check, such as one of the checks described inFIGS. 9-12). Next, the method650may return to the decision state658. If the decision state660determines there are no other classification tests, the method650may move to the state664.

The state664may activate a next one of the corrective measures152a-152n. Next, the method650may move to the decision state666. If the decision state666determines the obstructions156a-156ncannot be classified based on the response to the corrective measure activated, the method650may return to the state664. If the decision state666determines the obstructions156a-156ncan be classified based on the response to the corrective measure activated, the method650may move to the state668. The state668may activate one or more of the corrective measures152a-152nbased on the classification. Next, the method650may end at the state670.

The camera system100may perform one or more of the checks described inFIGS. 9-13. Other checks may be performed. The checks performed, the types of obstructions tested in each check and/or the changes in confidence level in response to each check may be varied according to the design criteria of a particular implementation. The checks may be performed to rule out and/or confirm the type of obstruction (e.g., increase and/or decrease a confidence level associated with one or more types of obstructions). Each of the individual checks or cues may not provide certainty for classifying a particular obstruction. Each of the checks may provide evidence that may be used to diagnose the cause of the reduction of visibility.

Results from each of the checks may be determined by the processor106and/or stored in the memory108(e.g., in the data storage portion118). The results from each of the checks may be aggregated by updating the stored information (e.g., to take into account various data received from each of the checks). In one example, the aggregated information may be a single value for each type of obstruction (e.g., a confidence level value). In another example, each type of obstruction may have many associated values based on the evidence received from the checks. The processing and/or storage of the aggregated results may be varied according to the design criteria of a particular implementation.

In aggregate, the evidence received in response to the checks may provide the processor106with enough information to classify the type of obstruction and/or activate a corrective measure. The response of the obstruction to the corrective measure may also provide feedback that may be used as evidence of the type of obstruction. For example, the obstruction may be continually and/or periodically monitored to classify the type of obstruction and/or monitored to ensure safe visibility. The corrective measures152a-152nmay be modified based on the feedback.

The particular corrective measures152a-152nthe vehicle50uses may depend on the particular problem determined (e.g., the classification of the type of obstruction). The particular corrective measure used may be discontinued as soon as visibility improves (e.g., as judged by computer vision process run on the processor106). In case of “foggy” glass, the processor106may constantly keep the windshield160at a temperature above the dew point. Since a feedback mechanism is implemented, the processor106may correct one or more assumptions regarding dew point based on whether the windshield160becomes foggy again. In an example when outside fog is detected, fog lights may be turned on automatically.

Detection and/or corrective measures may start to be applied as soon as the driver turns on the vehicle50. In one example, approaching the vehicle50(where the vehicle detects the key fob), inserting ignition key, using external app, etc., may start the detection and/or correction process (e.g., an activation signal). In some embodiments, automatically starting the process for a particular time of day and/or a schedule (e.g., before work so ice can be removed before the driver has to manually scrape the ice) may provide additional convenience to the driver.

The processor106may be used to differentiate between various types of obstructions (e.g., classify the obstructions156a-156n). For example, ice and/or condensation may differ in color histogram. In another example, condensation and/or fog may differ in the way they reflect headlights, respond to A/C activation, etc. The apparatus100may implement an integrated process that combines differentiating and/or attempting to fix various obstructions iteratively. For example, turning on the headlights briefly may be used to test the “hypothesis” that there is fog outside. In another example, spraying water on the windshield160may be a corrective measure used to test the “hypothesis” that there is ice formed on the windshield160(the water might be both the test and a part of the solution).

The corrective measures152a-152nmay be implemented as part of a “diagnosis” (e.g., classification of the obstructions156a-156n). The response of the obstructions156a-156nto the corrective measures152a-152nmay increase or decrease a confidence level of the classification. For example, applying heat to increase the temperature of the inside of the windshield160above the dew point may increase visibility. The response of the obstruction (e.g., the fog fading) to the corrective measure may increase the confidence level (e.g., provide a confirmation) of the classification.

The apparatus100may provide automatic cleaning, defogging and/or de-icing of windshields/windows based on visual information from cameras complemented by sensor data. In one example, fog lights may be turned on based on visual information from cameras complemented by sensor data. A feedback loop for detecting whether the problem is gone after applying a corrective measure and/or modifying the behavior accordingly in response to the corrective measures may be used to minimize distractions. The apparatus100may provide the classification of information to determine a speed of visibility deterioration.