Accelerometer-based external sound monitoring for backup assistance in a vehicle

Accelerometer-based external sound monitoring for backup assistance in a vehicle. The vehicle includes an accelerometer mounted on an internal surface of a rear window of the vehicle, a rear view camera, and a processor. The processor identifies a source of a sound that causes vibrations on the rear window that are captured by the accelerometer. The processor also categorizes the sound and provides an alert based on a category of the sound.

TECHNICAL FIELD

The present disclosure generally relates to back up assistance systems in a vehicle and, more specifically, accelerometer-based external sound monitoring for backup assistance in a vehicle.

BACKGROUND

Increasingly, vehicles are being manufactured with rear view cameras and ultrasonic sensors to provide assistance when the vehicle is reversing or about to reverse. Maintaining these systems can be difficult. For example, dirt or snow may block the camera or the ultrasonic sensors. Additionally, especially for the ultrasonic sensors, the angle at which the vehicle is currently traversing may cause objects behind the vehicle to be out of the line of sight and, thus, not detectable.

SUMMARY

An example vehicle includes an accelerometer mounted on an internal surface of a rear window of the vehicle, a rear view camera, and a processor. The processor identifies a source of a sound that causes vibrations on the rear window that are captured by the accelerometer. The processor also categorizes the sound and provides an alert based on a category of the sound.

An examples method to monitor an area behind a vehicle includes measuring signals with an accelerometer mounted on an internal surface of a rear window of the vehicle, the signals caused by sound waves vibrating the rear window. The method also includes identifying a source of a sound waves and categorizing the source. Additionally, the example includes providing an alert based on a category of the source.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Vehicles with rear view cameras and rear-facing ultrasonic sensors assist drivers detect obstacles behind the vehicle when the vehicle is in reverse. The images captured by the rear view camera are displayed on a display, often built into the center console of the vehicle. Some vehicles include an active safety module that identifies and/or categorizes the objects based on the images captured by the rear-view camera and/or the measurements by the ultrasonic sensors. However, the rear-view camera and ultrasonic sensors may become obscured. For example, dirt may accumulate on the camera or the ultrasonic sensors Additionally, objects behind the vehicle may be out of the line of sight of the rear-view camera and ultrasonic sensors because of obstacles to the side of the roadway obscuring the view (e.g., a hedge, a fence, a wall, etc.) or a change in the angle in the roadway causes the line of sight of the sensors to be angled upwards and thus not capture objects close to the ground (e.g., a bicycle, a pet, etc.).

As described below, a vehicle monitors and categorizes sounds external to the vehicle. However, traditional electric condenser microphones (ECMs) are not constructed for external use. Their delicate microphone diaphragms can be rendered unusable by dirt, snow, rain, or mud. Instead, because sound causes oscillation as it travels through a medium, accelerometers are placed on one or more of the edge of vehicle glass surfaces. This oscillation is measured as a vibration by the accelerometer when the pressure wave impinges on a window of the vehicle. The resulting electrical output from the accelerometer is processed by sound categorizer that categorizes into importance categories (e.g., a critical category and a non-critical category, etc.) and/or identifies the sound. The vehicle provides an alert based on the category and/or identity of the sound. For example, sounds indicative of human speech may be categorized as critical while sound indicative of average traffic noise may be categorized as non-critical. The accelerometers are mounted on one or more of the windows to the rear of the vehicle (e.g., the back glass, the left backseat door glass, the right backseat door glass, etc.). Additionally, in some examples, the sound categorizer operates in one of two modes. In a first mode (sometimes referred to as a “passive mode”), the sound categorizer causes a captured sound to play on a sound system of the vehicle when the sound is categorized as critical. In a second mode (sometimes referred to as an “active mode”), the sound categorizer causes a captured sound to play on a sound system of the vehicle regardless of the categorization of the sound.

FIG. 1illustrates a vehicle100operating in accordance with the teachings of this disclosure. The vehicle100may be a standard gasoline powered vehicle, a hybrid vehicle, an electric vehicle, a fuel cell vehicle, and/or any other mobility implement type of vehicle. The vehicle100includes parts related to mobility, such as a powertrain with an engine, a transmission, a suspension, a driveshaft, and/or wheels, etc. The vehicle100may be non-autonomous or semi-autonomous (e.g., some routine motive functions controlled by the vehicle100). Alternatively, in some examples, the vehicle100may be an autonomous vehicle (e.g., routine motive functions controlled by the vehicle100). In the illustrated example the vehicle100includes windows102, one or more accelerometers104, a powertrain control unit (PCU)106, an active safety module (ASM)108, and an infotainment head unit110.

The windows102include a windshield, a back glass, a left front door glass, a left backseat door glass, a right front door glass, a right backseat door glass, and/or a moon roof. Some of the windows102are made of laminated glass (sometimes referred to as “safety glass”) (e.g., the windshield, etc.) and some of the windows102are made of non-laminated tempered glass (e.g., the left front door glass, the left backseat door glass, the right front door glass, the right backseat door glass, etc.). The glass of the windows102vibrates when struck by sound waves.

The accelerometer(s)104may be any type of accelerometer that (a) measures the vibrations perpendicular to the plane of glass of the corresponding window102and (b) measures a wide frequency range (e.g., the frequency range of audible sound, etc.), including uniaxial or triaxial accelerometers, micromachined or piezoelectric accelerometers, etc. The accelerometer(s)104is/are rigidly mounted on one or more of the windows102on the interior of the cabin of the vehicle100. In the illustrated example, the accelerometers104is mounted on the back glass window. In some examples, to improve the arc behind the vehicle100in which the noise from obstacles is detected, the accelerometers104are installed on other windows102, such as the left backseat door glass and/or the right backseat door glass.

The powertrain control unit106includes hardware and firmware to control the ignition, fuel injection, emission systems, transmission and/or the brake system of the vehicle100. The powertrain control unit106monitors sensors (such as fuel injection sensors, wheel speed sensors, exhaust sensors, etc.) and uses control algorithms to control, for example, fuel mixture, ignition timing, variable cam timing, emissions control, a fuel pump, an engine cooling fan and/or a charging system. Additionally, the powertrain control unit106communicates statuses of the vehicle100relating to the powertrain (e.g., the status of the ignition, the status of the transmission, the speed of the vehicle100, etc.) onto a vehicle data bus (e.g., the vehicle data bus202ofFIG. 2below).

The active safety module108includes hardware and software to detect and identify objects proximate the vehicle100based on images captured by one or more cameras (e.g., a rear view camera112, etc.) and/or range detection sensors (e.g., the ultrasonic sensors114, etc.) using image recognition and spatial recognition techniques. The active safety module108categorizes the objects and, in some examples, requests a warning be provided to the driver based on the category of the detected object. The detection and identification of obstacles is use in various functions of the active safety module108, such as traffic sign recognition and/or collision avoidance, etc.

The infotainment head unit110provides an interface between the vehicle100and a user. The infotainment head unit110includes digital and/or analog interfaces (e.g., input devices and output devices) to receive input from the user(s) and display information. The input devices may include, for example, a control knob, an instrument panel, a digital camera for image capture and/or visual command recognition, a touch screen, an audio input device (e.g., cabin microphone), buttons, or a touchpad. The output devices may include instrument cluster outputs (e.g., dials, lighting devices), actuators, a heads-up display, a center console display (e.g., a liquid crystal display (“LCD”), an organic light emitting diode (“OLED”) display, a flat panel display, a solid state display, etc.), and/or speakers. In the illustrated example, the infotainment head unit110includes hardware (e.g., a processor or controller, memory, storage, etc.) and software (e.g., an operating system, etc.) for an infotainment system (such as SYNC® and MyFord Touch® by Ford®, Entune® by Toyota®, IntelliLink® by GMC®, etc.). Additionally, the infotainment head unit110displays the infotainment system on, for example, the center console display.

In the illustrated example, the infotainment head unit110includes database of sound signatures116and an obstacle monitor118to detect sounds and identify of objects behind the vehicle100and provides an audio, visual, and/or haptic warning to a driver when (a) the rear view camera112and/or the rear-facing ultrasonic sensors114are obscured or malfunctioning, and/or (b) the vehicle100is positioned such that the field of view119of the rear view camera112and/or the field of view121of the rear-facing ultrasonic sensors114cannot observe the area behind the vehicle100(e.g., the vehicle100is positioned on a hill, etc.). The obstacle monitor118processes signals produced by the accelerometer(s)104that are caused by sounds waves contacting the corresponding window(s)102. The obstacle monitor118compares the detected sound to signatures in the database of sound signatures116and categorizes the sound. Based on the category, the obstacle monitor118provides an audio, visual, and/or haptic alert. For example, the obstacle monitor118may cause the sound system of the infotainment head unit110to emit a chime and a steering wheel control module to vibrate the steering wheel. Additionally, in some examples, depending on a mode selected by a user, the obstacle monitor118plays the sound over speakers120of the vehicle100.

The database of sound signatures116contains signatures to be compared to the signals from the accelerometer104that are processed by the obstacle monitor118. As used herein, a signature is an identifiable segment of a signal in the time domain or the frequency domain that is associated with a source of sound. For example, a signature may associate a signal in the frequency domain to the sound indicative of human speech or an animal's cry. The database of sound signatures116may be any suitable data structure that associates a time or frequency domain representation of a sound to an event, phenomenon, and/or source of the sound. The database of sound signatures116associates each signature with a category and an identifier. In some examples, the database of sound signatures116associates each signature with a natural language identifier (e.g., “cat,” “traffic noise,” etc.) and/or a sound file of the nature language identifier to be played on the speakers120.

The obstacle monitor118determines whether the rear-view camera112and/or the rear-facing ultrasonic sensors114are obscured or malfunctioning (e.g., not providing data/images, etc.). In some examples, to determine whether the rear-view camera112is obscured (e.g., by snow or mud, etc.), the obstacle monitor118analyzes the average light intensity of the pixels of the images captured by the rear-view camera112. For example, the rear-view camera112is likely obscured when the average light intensity of the pixels is low. Alternatively or additionally, in some examples, to determine whether the rear-view camera112is obscured, the obstacle monitor118analyzes the light intensity of the pixels of the images captured by the rear-view camera112to determine whether a majority of the pixels have a substantially similar (e.g.,) pixel intensity.

The obstacle monitor118filters and processes, in the time and frequency domains, the signals caused by sound waves interacting with the window(s)102. The obstacle monitor118compares the processed signal to the signatures in the database of sound signatures116to identify the source of the signal and categorized the source of sound into categories of importance. For example, the obstacle monitor may categorizes a sound as critical or non-critical. A critical sound is a sound that originates from an object122that has a high likelihood of being behind the vehicle100or soon to be behind the vehicle100. For example, critical sounds may includes human sounds, animals sounds, horns (e.g., car horns, etc.) and/or bells (e.g., bicycle bells, etc.), etc. Non-critical sounds are sounds that are likely ambient or environmental sounds, such as music, traffic noise, rain, etc. In some examples, obstacle monitor118adjusts the category of the sound based on the intensity (e.g., in decibel-milliwatts (dBm)) of the sound as detected by the accelerometer104. For example, a sound that is associated with an object categorized as critical may be categorizes as non-critical if the intensity of the sound is less that a threshold. In such a manner, the obstacle monitor118distinguishes between objects122that are near the rear of the vehicle100and objects122that are close to the vehicle.

When the rear-view camera112and/or the rear-facing ultrasonic sensors114are not obscured, identities of objects122behind the vehicle100generated by the active safety module108are prioritized. That is, when the active safety module108identifies an object122in the critical category, the obstacle monitor118provides an audio, visual, and/or haptic alert associated with identity of the object122determined by the active safety module108. However, even then the identities of objects122behind the vehicle100generated by the active safety module108are prioritized, the obstacle monitor118still independently attempts to identify the source (e.g., the object122) of the sound. In an example scenario (e.g., as illustrated inFIG. 1), the vehicle100may be positioned on a hill such that the object122generating sound is outside the fields of view119and121of the rear-view camera112and the ultrasonic sensors114. In such an example scenario, the obstacle monitor118may provide an identity of the object122even when the non-obscured rear-view camera112and ultrasonic sensors114cannot observe the object122.

When the rear-view camera112and/or the rear-facing ultrasonic sensors114are obscured, the obstacle monitor118provides the identity of the sound to the active safety module108. Additionally, when the obstacle monitor118identifies the object122based on the sound detected by the accelerometer104that is in the critical category, the obstacle monitor118provides an audio, visual, and/or haptic alert associated in the database of sound signatures116with identity of the object122. In some examples, the alert includes the natural language identifier associated with the detected object122. For example, the obstacle monitor118may cause “Warning. A dog may be behind the vehicle” to play on the speakers120of the vehicle100when the obstacle monitor118determines that the object122is a dog. In some examples, the when the sound is categorized as critical, the obstacle monitor118instructs the powertrain control unit106to apply the brakes of the vehicle100to stop the motion of the vehicle100in reverse. That is, the obstacle monitor118causes the vehicle100to autonomously brake and ignore input of an acceleration pedal.

In some examples, the obstacle monitor118operates in different modes in which it reacts differently to the detected objects based on the category assigned to the object122. In some such examples, the obstacle monitor118operates in a “passive mode” and in an “active mode.” In the passive mode, the obstacle monitor118provides the alert and plays the sound detected by the accelerometer104on the speakers120when the object122is categorized as critical. When the obstacle monitor118operates in the active mode, the obstacle monitor118plays the sound detected by the accelerometer104on the speakers120regardless of the category associated with the object122. In some examples, the obstacle monitor118operates in a “stationary mode” in which the obstacle monitor118provides the alert and plays the sound detected by the accelerometer104on the speakers120when the vehicle100is stationary with the transmission in park. In such a manner, the obstacle monitor118provides alerts when, for example, the rear-view camera (and thus the back glass) is obscured by severe weather conditions (e.g., snow, etc.) and provides information for the occupants of the vehicle100to exit the vehicle100.

In some examples, when the vehicle100is an autonomous vehicle, the obstacle monitor118augment other autonomous functions of the vehicle100. For example, when one or more rear-facing sensors of the vehicle100are obscured, the autonomous vehicle may use the identified sounds as a source of information to facilitate parking in an emergency area. Additionally, in such examples, the vehicle100may provide information to the occupants of the vehicle100to exit the vehicle100.

FIG. 2is a block diagram of electronic components200of the vehicle100ofFIG. 1. In the illustrated example, the electronic components200include the accelerometer(s)104, the powertrain control unit106, the active safety module108, the infotainment head unit110, the rear view camera112, the ultrasonic sensors114, the speakers120, and a vehicle data bus202.

The infotainment head unit110includes a processor or controller204and memory206. In the illustrated example, the infotainment head unit110is structured to include obstacle monitor118. The processor or controller204may be any suitable processing device or set of processing devices such as, but not limited to: a microprocessor, a microcontroller-based platform, a suitable integrated circuit, one or more field programmable gate arrays (FPGAs), and/or one or more application-specific integrated circuits (ASICs). The memory206may be volatile memory (e.g., RAM, which can include non-volatile RAM, magnetic RAM, ferroelectric RAM, and any other suitable forms); non-volatile memory (e.g., disk memory, FLASH memory, EPROMs, EEPROMs, non-volatile solid-state memory, etc.), unalterable memory (e.g., EPROMs), read-only memory, and/or high-capacity storage devices (e.g., hard drives, solid state drives, etc). In some examples, the memory206includes multiple kinds of memory, particularly volatile memory and non-volatile memory. In the illustrated example, the memory206includes the database of sound signatures116.

The memory206is computer readable media on which one or more sets of instructions, such as the software for operating the methods of the present disclosure can be embedded. The instructions may embody one or more of the methods or logic as described herein. In a particular embodiment, the instructions may reside completely, or at least partially, within any one or more of the memory206, the computer readable medium, and/or within the processor204during execution of the instructions.

The vehicle data bus202communicatively couples the powertrain control unit106, the active safety module108, and the infotainment head unit110. In some examples, the vehicle data bus202includes one or more data buses. The vehicle data bus202may be implemented in accordance with a controller area network (CAN) bus protocol as defined by International Standards Organization (ISO) 11898-1, a Media Oriented Systems Transport (MOST) bus protocol, a CAN flexible data (CAN-FD) bus protocol (ISO 11898-7) and/a K-line bus protocol (ISO 9141 and ISO 14230-1), and/or an Ethernet™ bus protocol IEEE 802.3 (2002onwards), etc.

FIG. 3is a flowchart of a method to provide backup assistance with an accelerometer104mounted on back glass of the vehicle100ofFIG. 1, which may be implemented by the electronic components200ofFIG. 2. Initially, at block302, the obstacle monitor118waits until the ignition of the vehicle100is on. At block304, the obstacle monitor118waits until the transmission of the vehicle100in reverse. At block306, the obstacle monitor118determines the state (e.g., obscured, not obscured, etc.) of the rear-view camera112and/or the ultrasonic sensors114. In some examples, the obstacle monitor118analyzes the data captured by the rear-view camera112and/or the ultrasonic sensors114to determine whether the data is indicative of the rear-view camera112and/or the ultrasonic sensors114being obscured. For example, the obstacle monitor118may perform a pixel analysis on the images captured by the rear-view camera112to determine whether the light levels in the images are indicative of the rear-view camera112being obscured by snow. At block308, the obstacle monitor118determines wither the rear-view camera112and/or the ultrasonic sensors114is obscured or blocked. When the rear-view camera112and/or the ultrasonic sensors114is obscured of blocked, the method continues at block310. Otherwise, when the rear-view camera112and/or the ultrasonic sensors114is not obscured or blocked, the method continue at block316.

At block310, the obstacle monitor118prioritizes the obstacle detection based on the sounds detected by the accelerometer104mounted on the back glass of the vehicle100. At block312, the obstacle monitor118determines whether the accelerometer104detects a sound. When the accelerometer104detects a sound, the method continues at block314. Otherwise, when the accelerometer104does not detect a sound, the method returns to block302. At block314, the obstacle monitor118classifies the sound detected by the accelerometer and assigns the sound a category. In some examples, the obstacle monitor118filters and processes (e.g., in the time domain, in the frequency domain) the sound and compares it to signatures in the database of sound signatures116.

At block316, the obstacle monitor118prioritizes the obstacle detection based on analysis by the active safety module108. At block318, the obstacle monitor118determines whether the accelerometer104detects a sound. When the accelerometer104detects a sound, the method continues at block314. Otherwise, when the accelerometer104does not detect a sound, the method continues at block320. At block320, the obstacle monitor118does not provide an alert.

At block322, the obstacle monitor118determines whether it is in the passive mode. When the obstacle monitor118is in the passive move, the method continues at block324. Otherwise, when the obstacle monitor118is in the active mode, the method continues at block330. At block324, the obstacle monitor118determines whether the sound was categorized as critical at block314. When the sound was categorized as critical, the method continues at block326. Otherwise, when the sound was not categorized as critical, the method returns to block302. At block326, the obstacle monitor118provides an audio, visual, and/or haptic alert to the driver. At block328, the obstacle monitor118plays the sound through the speakers120of the sound system. At block330, the obstacle monitor118determines whether the sound was categorized as critical at block314. When the sound was categorized as critical, the method continues at block326to provide an audio, visual, and/or haptic alert to the driver. Otherwise, when the sound was not categorized as critical, the method continues at block328to play the sound through the speakers120of the sound system.

The flowchart ofFIG. 3is representative of machine readable instructions stored in memory (such as the memory206ofFIG. 2) that comprise one or more programs that, when executed by a processor (such as the processor204ofFIG. 2), cause the infotainment head unit110to implement the example obstacle monitor118ofFIGS. 1 and 2. Further, although the example program(s) is/are described with reference to the flowchart illustrated inFIG. 3, many other methods of implementing the example obstacle monitor118may alternatively be used. For example, the order of execution of the blocks may be changed, and/or some of the blocks described may be changed, eliminated, or combined.

An example vehicle includes an accelerometer mounted on an internal surface of a rear window of the vehicle, a rear view camera, and a processor. The processor identifies a source of a sound that causes vibrations on the rear window that are captured by the accelerometer. The processor also categorizes the sound and provides an alert based on a category of the sound.

In some examples, the processor determines whether the rear view camera is obscured, and when the rear view camera is obscure, provides the identity of the source to other vehicle subsystems. In some such examples, to determine whether the rear view camera is obscured, the processor performs a pixel level analysis of light intensities of images captured by the rear view camera. In some examples, memory stores a database of sound signatures. To identify the source of the sound, the processor processes the sound to generate an observed signature; and compares the observed signature to signatures stored in the database of sound signatures. In some examples, the processor categorizes the sound as either a first category or a second category. In some such examples, the processor provides the alert when the sound is assigned to the first category and does not provide the alert when the sound is assigned to the second category. In some such examples, the processor determines a mode of operation of the vehicle. In some such examples, when the vehicle is operating in a first mode, the processor plays the sound through speakers of the vehicle when the sound is categorized in the first category. In some such examples, when the vehicle is operating is a second mode, the processor plays the sound through the speakers of the vehicle regardless of how the sound is categorized.

An examples method to monitor an area behind a vehicle includes measuring signals with an accelerometer mounted on an internal surface of a rear window of the vehicle, the signals caused by sound waves vibrating the rear window. The method also includes identifying a source of a sound waves and categorizing the source. Additionally, the example includes providing an alert based on a category of the source.

In some example, to determine whether the rear view camera is obscured, the method includes performing a pixel level analysis of light intensities of images captured by the rear view camera. In some examples, to identify and categorize the source, the method includes processing the signals measured by the accelerometer to generate an observed signature and comparing the observed signature to signatures stored in the database of sound signatures. IN some examples, the method includes providing the alert when the sound is assigned to a first category and not providing the alert when the sound is assigned to a second category. In some examples, the method includes (a) when the vehicle is operating in a first mode, playing the sound waves through speakers of the vehicle when the sound is categorized in a first category, and (b) when the vehicle is operating in a second mode, playing the sound through the speakers of the vehicle regardless of how the sound is categorized.

In this application, the use of the disjunctive is intended to include the conjunctive. The use of definite or indefinite articles is not intended to indicate cardinality. In particular, a reference to “the” object or “a” and “an” object is intended to denote also one of a possible plurality of such objects. Further, the conjunction “or” may be used to convey features that are simultaneously present instead of mutually exclusive alternatives. In other words, the conjunction “or” should be understood to include “and/or”. As used here, the terms “module” and “unit” refer to hardware with circuitry to provide communication, control and/or monitoring capabilities, often in conjunction with sensors. “Modules” and “units” may also include firmware that executes on the circuitry. The terms “includes,” “including,” and “include” are inclusive and have the same scope as “comprises,” “comprising,” and “comprise” respectively.