System and method to provide a misplacement notification

One general aspect includes a system to provide a misplacement notification, the system including: a memory configured to include a program and a processor configured to execute the program, where the program enables the processor to, after a vehicle ingress event, cause a sensor to capture a reference image of a portion of a vehicle interior; after a vehicle egress event, cause the sensor to capture a test image of the portion of the vehicle interior; determine whether the test image includes one or more objects not found in the reference image; and generate a notification, based on the determination of whether the test image includes one or more objects not found in the reference image.

INTRODUCTION

It is easy for vehicle drivers to put items in the backseat of their vehicle at the beginning of a trip and forget to remove these items after reaching their final destination. In the vehicle-share environment, once the items are left behind, it is nearly impossible for the past driver to reclaim their belongings. When a child or animal is left behind, such an incident can pose health and safety issues for the unattended child or animal. Accordingly, it is desirable to provide a system and method that will provide a misplacement notification when an item has been left in the backseat of a vehicle. It is also desirable for this system and method to further take one or more precautionary steps to protect a forgotten child, infant, or animal. Furthermore, other desirable features and characteristics of the present invention will become apparent from the subsequent detailed description of the invention and the appended claims, taken in conjunction with the accompanying drawings and this background of the invention.

SUMMARY

A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions. One general aspect includes a system to provide a misplacement notification, the system including: a memory configured to include a program and a processor configured to execute the program, where the program enables the processor to, after a vehicle ingress event, cause a sensor to capture a reference image of a portion of a vehicle interior; after a vehicle egress event, cause the sensor to capture a test image of the portion of the vehicle interior; determine whether the test image includes one or more objects not found in the reference image; and generate a notification, based on the determination of whether the test image includes one or more objects not found in the reference image. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

Implementations may include one or more of the following features. The system further including, after the vehicle ingress event, cause one or more vehicle interior lights to illuminate the vehicle interior in support of capturing the reference image; and after the vehicle egress event, cause the one or more vehicle interior lights to illuminate the vehicle interior in support of capturing the test image. The system further including where, when it is determined that the test image includes one or more objects not found in the reference image, access an image recognition database stored in the memory to recognize the one or more objects included in the test image; and release a vehicle door or contact an emergency services provider or some combination thereof only when the one or more objects included in the test image but not found in the reference image are recognized as a certain type. The system where the one or more objects included in the test image are recognized as an animal, child, or infant. The system where the notification is a text message configured to be exhibited on an interface of a mobile computing device. The system where the notification is an at least temporary activation of a vehicle horn system. The system where the portion of the vehicle interior is a seat cushion of a backseat located in the vehicle interior. The system where the vehicle ingress event is a vehicle ignition being turned to an ON state. The system where the vehicle ingress event includes a vehicle ignition being turned to an OFF state, a vehicle door being temporarily opened, and the vehicle door being locked. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

One general aspect includes a method to provide a misplacement notification, the method including, after a vehicle ingress event, via a controller, causing a sensor to capture a reference image of a portion of a vehicle interior; after a vehicle egress event, via the controller, causing the sensor to capture a test image of the portion of the vehicle interior; determining, via the controller, whether the test image includes one or more objects not found in the reference image; and generating a notification, via the controller, based on the determination of whether the test image includes one or more objects not found in the reference image. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

Implementations may include one or more of the following features. The method further including, after the vehicle ingress event, via the controller, causing one or more vehicle interior lights to illuminate the vehicle interior in support of capturing the reference image; and after the vehicle egress event, via the controller, causing the one or more vehicle interior lights to illuminate the vehicle interior in support of capturing the test image. The method further including: where, when it is determined that the test image includes one or more objects not found in the reference image, via the controller, accessing an image recognition database to recognize the one or more objects included in the test image; and releasing a vehicle door or contacting an emergency services provider or some combination thereof, via the controller, only when the one or more objects included in the test image but not found in the reference image are recognized as a certain type. The method where the one or more objects included in the test image are recognized as an animal, child, or infant. The method where the notification is a text message configured to be exhibited on an interface of a mobile computing device. The method where the notification is an at least temporary activation of a vehicle horn system. The method where the portion of the vehicle interior is a seat cushion of a backseat located in the vehicle interior. The method where the vehicle ingress event is a vehicle ignition being turned to an ON state. The method where the vehicle ingress event includes a vehicle ignition being turned to an OFF state, a vehicle door being temporarily opened, and the vehicle door being locked. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

One general aspect includes a vehicle including a body and a vehicle door adapted to releasably engage the body; an interior including a backseat having a seat cushion and one or more vehicle interior lights configured to illuminate the vehicle interior; a camera positioned in the interior, the camera configured to capture an image of at least the seat cushion of the backseat; a memory that is disposed onboard the vehicle and configured to include a program; a processor that is disposed onboard the vehicle and configured to execute the program, where the program enables the processor to: after a vehicle ignition has been turned to an ON state, cause the one or more vehicle interior lights to illuminate the vehicle interior and then cause the camera to capture a reference image of at least the seat cushion of the backseat; after the vehicle ignition being turned to an OFF state, the vehicle door being temporarily opened, and the vehicle door being locked, cause the one or more vehicle interior lights to illuminate the vehicle interior and then cause the camera to capture a test image of at least the seat cushion of the backseat; determine whether the test image includes one or more objects not found in the reference image; and generate a text message configured to be exhibited on a interface of a mobile computing device or at least temporary activate a vehicle horn system or some combination thereof, based on the determination of whether the test image includes one or more objects not found in the reference image. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

Implementations may include one or more of the following features. The vehicle further including: where, when it is determined that the test image includes one or more objects not found in the reference image, access an image recognition database stored in the memory to recognize the one or more objects included in the test image; and release the vehicle door or contact an emergency services provider or some combination thereof only when the one or more objects included in the test image but not found in the reference image are recognized as an animal, child, or infant. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and is not intended to limit the application and uses. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding background and brief summary or the following detailed description. As used herein, the term module refers to an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that executes one or more software or firmware programs or code segments, a combinational logic circuit, and/or other suitable components that provide the described functionality.

As shown inFIG. 1, there is shown a non-limiting example of a communication system10that may be used together with examples of the system disclosed herein and/or to implement examples of the methods disclosed herein. Communication system10generally includes a vehicle12, a wireless carrier system14, a land network16, a data center18(i.e., the backend), and an emergency services provider75. It should be appreciated that the overall architecture, setup and operation, as well as the individual components of the illustrated system are merely exemplary and that differently configured communication systems may also be utilized to implement the examples of the system and/or method disclosed herein. Thus, the following paragraphs, which provide a brief overview of the illustrated communication system10, are not intended to be limiting.

Vehicle12may be any type of manually operated or autonomous vehicle such as a motorcycle, car, sports utility vehicle (SUV), truck, bus, bicycle, recreational vehicle (RV), construction vehicle (e.g., bulldozer), train, trolley, marine vessel (e.g., a boat), aircraft (e.g., airplane, helicopter, etc.), amusement park vehicle, farm equipment, golf cart, etc., and is equipped with suitable hardware and software that enables it to communicate over communication system10. In certain embodiments, vehicle12may include a power train system with multiple generally known torque-generating devices including, for example, an engine. The engine may be an internal combustion engine that uses one or more cylinders to combust fuel, such as gasoline, in order to propel vehicle12. The power train system may alternatively include numerous electric motors or traction motors that convert electrical energy into mechanical energy for propulsion of vehicle12.

Some of the fundamental vehicle hardware20is shown generally inFIG. 1including a telematics unit24, a microphone26, speaker28, and buttons and/or controls30connected to telematics unit24. Operatively coupled to telematics unit24is a network connection or vehicle bus32. Examples of suitable network connections include a controller area network (CAN), WIFI, Bluetooth, and Bluetooth Low Energy, a media oriented system transfer (MOST), a local interconnection network (LIN), a local area network (LAN), and other appropriate connections such as Ethernet or those that conform with known ISO (International Organization for Standardization), SAE (Society of Automotive Engineers), and/or IEEE (Institute of Electrical and Electronics Engineers) standards and specifications, to name a few.

The telematics unit24can be an OEM-installed (embedded) or aftermarket communication system which provides a variety of services through its communications with the data center18, and generally includes an electronic processing device38, one or more types of electronic memory40, a cellular chipset/component34, wireless modem36, an antenna system70including one or more antennas, and navigation unit containing a GPS chipset/component42capable of communicating location information via a GPS satellite system69. GPS component42thus receives coordinate signals from a constellation of GPS satellites65. From these signals, the GPS component42can determine vehicle position, which may be used for providing navigation and other position-related services to the vehicle operator. Navigation information can be presented on a display of telematics unit24(or other display within the vehicle) or can be presented verbally such as is done when supplying turn-by-turn navigation. The navigation services can be provided using a dedicated in-vehicle navigation module (that can be part of GPS component42), or some or all navigation services can be done via telematics unit24, wherein the location coordinate information is sent to a remote location for purposes of providing the vehicle with navigation maps, map annotations, route calculations, and the like.

The telematics unit24may provide various services including: turn-by-turn directions, map-based directions, and other navigation-related services provided in conjunction with the GPS component42; airbag deployment notification and other emergency or roadside assistance-related services provided in connection with various crash and/or collision sensor interface modules66and collision sensors68located throughout the vehicle and/or infotainment-related services where music, internet web pages, movies, television programs, videogames, and/or other content are downloaded by an infotainment center46operatively connected to the telematics unit24via vehicle bus32and audio bus22. In one example, downloaded content is stored for current or later playback. The above-listed services are by no means an exhaustive list of all the capabilities of telematics unit24, but are simply an illustration of some of the services telematics unit24may be capable of offering. It is anticipated that telematics unit24may include a number of additional components in addition to and/or different components from those listed above.

Vehicle communications may use radio transmissions to establish a communication channel (voice channel and/or data channel) with wireless carrier system14so that both voice and/or data transmissions can be sent and received over the channel. Vehicle communications are enabled via the cellular component34for voice communications and the wireless modem36for data transmission. Any suitable encoding or modulation technique may be used with the present examples, including digital transmission technologies, such as TDMA (time division multiple access), CDMA (code division multiple access), W-CDMA (wideband CDMA), FDMA (frequency division multiple access), OFDMA (orthogonal frequency division multiple access), etc. To accomplish this effect, dual mode antenna70services the GPS component42and the cellular component34.

Microphone26provides the driver or other vehicle occupant with a means for inputting verbal or other auditory commands, and can be equipped with an embedded voice processing unit utilizing a human/machine interface (HMI) technology known in the art. Conversely, speaker28provides audible output to the vehicle occupants and can be either a stand-alone speaker specifically dedicated for use with the telematics unit24or can be part of a vehicle audio component64. In either event, microphone26and speaker28enable vehicle hardware20and data center18to communicate with the occupants through audible speech. The vehicle hardware also includes one or more buttons and/or controls30for enabling a vehicle occupant to activate or engage one or more of the vehicle hardware components20. For example, one of the buttons and/or controls30can be an electronic pushbutton used to initiate voice communication with data center18(whether it be a human such as advisor58or an automated call response system). In another example, one of the buttons and/or controls30can be used to initiate emergency services.

The audio component64is operatively connected to the vehicle bus32and the audio bus22. The audio component64receives analog information, rendering it as sound, via the audio bus22. Digital information is received via the vehicle bus32. The audio component64provides amplitude modulated (AM) and frequency modulated (FM) radio, satellite radio, compact disc (CD), digital video disc (DVD), and multimedia functionality independent of the infotainment center46. Audio component64may contain a speaker system (which can be internally located in the vehicle interior or located on the exterior body of the vehicle12) or may utilize speaker28(which can also be internally or externally located on the vehicle12) via arbitration on vehicle bus32and/or audio bus22.

A camera63is operatively connected to the vehicle bus32. The camera63can be of the wide-angle or ultra-wide-angle variety and may be installed at various locations within an interior cabin of the vehicle12such as, for example, on the rear-view mirror, at some location on the ceiling of the vehicle interior, or on a seat backrest. The camera63can be angled such that second or third row seating (back seat) within in the interior cabin17(FIGS. 3A and 3B) is contained in the camera's63captured images. For example, one or more portions of the captured images may include the surfaces created by the seat cushions of these second or third row backseats. Skilled artisans will see that the camera63could also be angled such that the front row seats or various other locations within the cabin are contained in the camera's63captured images.

The vehicle crash and/or collision detection sensor interface66is operatively connected to the vehicle bus32. The collision sensors68provide information to telematics unit24via the crash and/or collision detection sensor interface66regarding the severity of a vehicle collision, such as the angle of impact and the amount of force sustained.

Vehicle sensors72, connected to various vehicle sensor modules44(VSMs) in the form of electronic hardware components located throughout vehicle12and use the sensed input to perform diagnostic, monitoring, control, reporting and/or other functions. Each of the VSMs44is preferably connected by vehicle bus32to the other VSMs, as well as to the telematics unit24, and can be programmed to run vehicle system and subsystem diagnostic tests. As examples, one VSM44can be an engine control module (ECM) that controls various aspects of engine operation such as fuel ignition and ignition timing. According to one embodiment, the ECM is equipped with on-board diagnostic (OBD) features that provide myriad real-time data, such as that received from various sensors including vehicle emissions sensors, fuel diagnostics sensors, and vehicle oil pressure sensors as well as provide a standardized series of diagnostic trouble codes (DTCs) which allow a technician to rapidly identify and remedy malfunctions within the vehicle. Another VSM44can be a body control module (BCM) that monitors and governs various electrical components located throughout the vehicle body like the vehicle's power door locks, power windows, air conditioner, tire pressure, lighting system, engine ignition, vehicle seat adjustment heating, mirrors, headlights, and horn system (which is configured to generate various audible warnings—chirps—as is generally known). Furthermore, as can be appreciated by skilled artisans, the above-mentioned VSMs are only examples of some of the modules that may be used in vehicle12, as numerous others are also possible.

A passive entry passive start (PEPS) module44, for instance, is another of the numerous of VSMs44and provides passive detection of the absence or presence of a passive physical key or a virtual vehicle key. When the passive physical key approaches, the PEPS module can determine if the passive physical key is authentic as belonging to the vehicle12. The PEPS can likewise use authentication information received from data center18to determine if a mobile computing device57with virtual vehicle key is authorized/authentic to vehicle12. If the virtual vehicle key is deemed authentic, the PEPS can send a command to BCM44permitting access to the vehicle12. It should be understood that the PEPS may be an electronic hardware component connected to the vehicle bus32or, in an alternative embodiment, may be one or more software code segments uploaded to electronic memory40.

Wireless carrier system14may be a cellular telephone system or any other suitable wireless system that transmits signals between the vehicle hardware20and land network16. According to an example, wireless carrier system14includes one or more cell towers48(only one shown), one or more cellular network infrastructures (CNI) (not shown), as well as any other networking components required to connect wireless carrier system14with land network16.

Land network16can be a conventional land-based telecommunications network connected to one or more landline telephones, and that connects wireless carrier system14to data center18as well as emergency services provider75(i.e., a fire department, hospital or police station having uniformed or otherwise identified employees or contractors). For example, land network16can include a public switched telephone network (PSTN) and/or an Internet protocol (IP) network, as is appreciated by those skilled in the art. Of course, one or more segments of the land network16can be implemented in the form of a standard wired network, a fiber or other optical network, a cable network, other wireless networks such as wireless local networks (WLANs) or networks providing broadband wireless access (BWA), or any combination thereof.

As revealed above, one of the networked devices that can directly or indirectly communicate with the telematics unit24is a mobile computing device57, such as (but not limited to) a smart phone, personal laptop computer or tablet computer having two-way communication capabilities, a wearable computer such as (but not limited to) a smart watch or glasses, or any suitable combinations thereof. The mobile computing device57can include computer processing capability, a transceiver53capable of communicating with remote locations (e.g., data center18), digital camera55, user interface59, and/or GPS module67capable of receiving GPS satellite signals and generating GPS coordinates based on those signals. User interface59may be embodied as a touch-screen graphical interface capable of user interaction as well as exhibiting information. Digital camera55may include the ability to generate bitmapped data representations of captured tangible object images through generally known operations. Examples of the mobile computing device57include the IPHONE™ and APPLE WATCH™ each being manufactured by Apple, Inc. and the GALAXY™ smart phone manufactured by Samsung Electronics Company as well as others.

Mobile device57may be used inside or outside of a vehicle, and may be coupled to the vehicle by wire or wirelessly. Mobile device57may also be configured to provide services according to a subscription agreement with a third-party facility or wireless/telephone service provider. It should be appreciated that various service providers may utilize the wireless carrier system14and that the service provider of telematics unit24may not necessarily be the same as the service provider of mobile device57.

When using a short-range wireless connection (SRWC) protocol (e.g., Bluetooth Low Energy, Wi-Fi, etc.), mobile computing device57and telematics unit24may pair with each other (or link to one another) on a case-by-case basis and while within a wireless range; SRWC pairing is known to skilled artisans. The SRWC protocol may be an aspect of telematics unit24or may be part of one or more independent VSMs44such as the PEPS and/or BCM44. Once SRWC is established, the devices may be considered bonded (i.e., they may recognize one another and/or connect automatically when they are in a predetermined proximity or range of one other. In other words—they may become, at least temporarily, network participants).

This unique pairing, for example, allows mobile computing device57to act as the virtual key fob briefly mentioned above. To illustrate how this occurs—upon receiving a request, data center18will generate an encrypted virtual vehicle key to permit vehicle access via mobile computing device57. Data center18will then transmit aspects this encrypted virtual vehicle key information to both mobile computing device57and the PEPS module44via telematics unit24. After paring has been established, mobile computing device57will send its virtual vehicle key aspect to telematics unit24for recognition in light of its stored corresponding virtual key aspect and in turn the PEPS module may establish mobile computing device57as the acting key fob for vehicle12. Data center18may also transmit one or more time parameters with the encrypted virtual vehicle key information so as to temporarily establish the virtual vehicle key of mobile device57.

Data center18is designed to provide the vehicle hardware20with a number of different system backend functions and, according to the example shown here, generally includes one or more switches52, servers54, databases56, advisors58, as well as a variety of other telecommunication/computer equipment60. These various data center components are suitably coupled to one another via a network connection or bus62, such as the one previously described in connection with the vehicle hardware20. Switch52, which can be a private branch exchange (PBX) switch, routes incoming signals so that voice transmissions are usually sent to either advisor58, or an automated response system, and data transmissions are passed on to a modem or other piece of telecommunication/computer equipment60for demodulation and further signal processing. The modem or other telecommunication/computer equipment60may include an encoder, as previously explained, and can be connected to various devices such as a server54and database56. Although the illustrated example has been described as it would be used in conjunction with a manned data center18, it will be appreciated that the data center18can be any central or remote facility, manned or unmanned, mobile or fixed, to or from which it is desirable to exchange voice and data.

Server54can incorporate a data controller which essentially controls its operations. Server54may control data information as well as act as a transceiver to send and/or receive the data information (i.e., data transmissions) from one or more of the databases56, telematics unit24, and mobile computing device57. The controller is moreover capable of reading executable instructions stored in a non-transitory machine readable medium and may include one or more from among a processor, microprocessor, central processing unit (CPU), graphics processor, Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), state machines, and a combination of hardware, software, and firmware components.

Database56could be designed to store information in the form of executable instructions such as, but not limited to, one or more application program interface (API) suites. One API suite can incorporate numerous vehicle-share services records (i.e., vehicle reservation information) each having information related to vehicle12such as, but not limited to, vehicle-share vehicle records (e.g., vehicle VSM information, vehicle system verification information/alerts, vehicle anomaly information), information related to the user such as, but not limited to, reservation account records (e.g., vehicle comfort settings information, driving record information, telematics unit settings, or vehicle make-model preferences), and information related to organizing vehicle reservations as well as fleet management such as, but not limited to, reservation profile records (e.g., reservation calendar information, vehicle assignment information, parking information, third party contact information, etc.); or any other pertinent vehicle-share system information. These records could moreover be written in SQL as well as be copied, organized, and/or stored in a tabular form to allow for continuous, real-time updates. The records may be accessible to the user, data center18, or one or more third parties. The vehicle-share records can additionally collaborate with a reservation account (discussed below) for support of, for example, reservation management and fleet management.

The user of mobile computing device57may create their own personalized vehicle reservation account to be stored in mobile memory61and which may have access to the vehicle-share records at the backend. The user may perform tasks to create this account through a variety of frontend devices such as, for example, through a remote computer and mobile computing device57. This reservation account may be uploaded to or accessible on server54(i.e., to support backend functions). Data center18may also access one or more additional remote servers and/or remote databases (e.g., Department of Motor Vehicles, weather databases, traffic databases, etc.) to receive information in support of establishing the reservation account as well as a particular reservation and one or more vehicle-share services records.

The reservation account may include validating data to verify and/or validate that future login attempts are secure (e.g., granting access only to the user). The validating data may include an account username and account password as well as user information (e.g., driver's license information), mobile computing device information such as, for example, the unique mobile device identifier (i.e., serial number). The user account may additionally store a variety of user preferences.

The user of mobile device57may visit an online software application store or web-service and download the reservation account therefrom. The reservation account may moreover include one or more prompts to instruct the user to provide information (e.g., validating data) to support account creation. Reservation account may also provide one or more prompts to assist a vehicle-share system user in reserving a fleet vehicle by operatively accessing and communicating with the backend vehicle-share services records.

As mentioned above, emergency services provider75can be an emergency services dispatch for a hospital, police station, fire department, or some other type of emergency medical technician group. As follows, the emergency services provider75has uniformed or otherwise identified employees or contractors that are specifically trained to rescue helpless victims from unfortunate situations. For example, in response to an emergency request, employees/contractors of the emergency services provider75can rescue helpless animals, children, or infants from being trapped in the interior cabin17of the otherwise assumingly unoccupied vehicle12. In order to rescue these helpless victims, the employees/contractors may have to use tools to unlock vehicle door13or they may have to contact live advisor58to remotely unlock the vehicle door13.

Method

Now turning toFIG. 2, there is shown an embodiment of a method200for generating and transmitting a misplacement notification when one or more objects have been unintentionally left in the backseat of a vehicle. One or more aspects of notification method200may be completed through telematics unit24which may include an electronic processing device38(processor) to execute one or more programs contained electronic memory40. One or more aspects of method200may also be completed by data center18, for example, via server54accessing an image recognition database stored in databases56. One or more ancillary aspects of method200may be completed by camera(s)63(sensor), audio component64, door13, and vehicle interior light15(FIG. 3A). Method200is supported by the PEPS module44being configured to establish a short range wireless communication protocol (SRWC protocol) with a virtual vehicle key fob when the mobile computing device57is within proximity of vehicle12. Method200is also supported by the BCM44being configured to cause the power door locks to unlock the door13as well as release the door latch and cause the door to open. Method200is also supported by audio system64being configured to generate an audible warning. Method200is also supported by the BCM44being configured to cause the horn system to generate an audible warning, unlock the vehicle doors, and open the power windows. These configurations may be established by a manufacturer at or around the time of the vehicle's assembly. The method200is further yet supported by preconfiguring infotainment module to exhibit information such as notifications on its graphics display.

Method200begins at201in which the user comes into proximity of vehicle12so as to establish a SRWC between one or more of the vehicle modules44(e.g., PEPS modules) and a mobile computing device57(having a virtual key fob stored thereon) or physical key fob on the person of the user. Moreover, the method begins when the PEPs module44determines that the physical/virtual key fob is authentic as belonging to vehicle12and the PEPs module permits at least temporary access to vehicle12via the mobile computing device57or physical key fob. In the vehicle-share system scenarios, method200may also begin at the start of a vehicle reservation.

In step210, the user triggers a vehicle ingress event. In various embodiments, this event occurs when the user unlocks the vehicle door13via their virtual/physical key fob, gets into the vehicle12and subsequently turns the vehicle ignition to the ON state (i.e., starting the vehicle ignition in order to operate the vehicle).

In step220, upon the ignition being started, telematics unit24will immediately wake up camera63(e.g., installed on the rearview mirror, at some location on the interior ceiling, or on the backside of the backrest of a vehicle seat). As such, power from the vehicle's battery (not shown) will be temporarily provided to camera63so it can be activated and capable of capturing one or more images. In this step, in various embodiments, a light sensor122may detect whether it is day or night in the vehicle environment. When it is determined that it is night outside, power may also be provided to the vehicle interior light15to illuminate the interior cabin17and help minimize lighting differences in the interior cabin.

In step230, telematics unit24will activate the camera63. With additional reference toFIG. 3A, the telematics unit24will moreover cause the camera63to capture an image301of, for example, of one or more seat cushions of the backseat19. In various embodiments, the telematics unit24may alternatively or simultaneously activate a second camera63′ installed elsewhere in the interior17to capture a supporting image of the same seat cushion(s) for the purposes of accuracy. When it is night outside of the vehicle12, the illuminated interior17will also support the visibility of the seat cushion as well as the rest of the backseat19. It should be understood that even though a second-row backseat19is shown, the camera63may also capture a front row seat as well as a third or fourth-row backseat19in certain types of vehicles12(in such instances, the camera(s)63will be placed around the vehicle interior17accordingly). It should also be understood that this captured image301or images should be considered a “reference image” or “reference images.”

In step240, the user operates the vehicle for some duration of time. For example, the user drives the vehicle to a grocery store.

In step250, with additional reference toFIG. 3B, after the user arrives at their destination (e.g., the grocery store) the user triggers a vehicle egress event. In various embodiments, this event occurs when the user puts the vehicle12in park, turns the vehicle ignition to an OFF state (i.e., powering down the vehicle ignition in order to cease vehicle operations), opens the vehicle door13, gets out of the vehicle, closes the vehicle door13behind them, and then locks the vehicle door13. Alternatively, in vehicle-share scenarios, the egress event may additionally be triggered when the vehicle reservation is complete.

Moreover, in this step, telematics unit24will reactivate the camera63(previously activated in step230) and capture a second image302of the locations of the vehicle interior17previously captured with the reference image(s). For example, the camera63will capture a second image of the one or more seat cushions of the same, previously captured backseat(s)19. In various embodiments, telematics unit24may alternatively or simultaneously activate another interior camera63′ to capture a supporting image of the seat cushion for accuracy purposes. The camera63may also be adjusted to the settings which were defined when the camera63first captured the reference image301. It should be understood that this second captured image302(or second set of captured images) should be considered a “test image” or “test images.” In certain embodiments, when it is nighttime and prior to the reactivation of camera63, power may also be provided to the vehicle interior light15to illuminate the interior cabin17, to minimize lighting differences, and to provide visibility of the seat cushion as well as the rest of the backseat19and in support of properly capturing the test image.

In step260, in various embodiments, telematics unit24will transmit the reference image and test image to data center18. In step270, server54will perform one or more generally known object detection techniques to determine whether an object has been left on the backseat of the vehicle interior17. For example, controller128can perform this image analysis by comparing the test image302(FIG. 3B) to the reference image301(FIG. 3A) at the pixel level to determine if the test image includes an object not found in the reference image. Thus, since the scene in the captured images should be static and the images are captured somewhat close in time, the reference and test images should be practically identical unless objects are unintentionally left in back of the vehicle cabin17. If such objects are found to be resting on the cushion of the backseat19, then the pixels310in the area(s) where the objects are found will have different characteristics such as, for example, different color(s) and intensity. As follows, when a certain number of pixels310are found to have unique characteristics not seen in the reference image301(e.g., on the backseat cushion19), for example—50 pixels, these differences would indicate one or more objects have been accidentally left behind. Furthermore, if the test image302includes one or more objects not found in the reference image, method200can move to optional step280; otherwise, method200will move to completion202. It should be understood that only certain pixel clusters310can be seen in the reference image301and test image302, however, skilled artists will understand that the entire area of these images should comprise pixels.

In optional step280, server54performs a generally known object recognition technique (WATSON VISUAL RECOGNITION™ by IBM or TENSOFLOW™ by GOOGLE) to recognize the objects included in the test image. For instance, server54can access an image recognition database from databases56(a forgotten object recognition API) filled with training data that can be a number of sample images depicting various kinds of objects which may accidentally be left behind in a vehicle (e.g., groceries, sports equipment, animals, children, plants, etc.). Moreover, server54can then search this database for a sample image that depicts an object of similar characteristics (size, color, pattern, shape, etc.) to at least one of the unique objects discovered in the test image. If the search uncovers a sample image including an object of identical or substantially similar characteristics to the object found in the test image, then the object in the test image will be recognized as the object associated with the sample image. The object in the test image may also be provided a percentage designation to indicate how certain the server54is that the object falls within the category of similar objects found in the database. After such a determination, server54may also provide the test image302to this image recognition database, to be added to the training data. Once the object in the test image302has been recognized, method200moves to step290.

If the search cannot find a sample image with an object like the one found in the test image302, server54may transmit the test image302to mobile computing device57(e.g., the smart phone associated with the user's reservation account) as a notification displayed on the user interface59, so as to let the user of mobile computing device57know something may have been forgotten in the vehicle12. This test image notification may also be accompanied by a visible text message also displayed on the user interface59. For example, this supporting text message may state: “WARNING, PLEASE REVIEW THE ACCOMPANYING IMAGE CAREFULLY TO MAKE SURE THE OBJECT IN THIS IMAGE IS NOT AN ITEM ACCIDENTALLY LEFT BEHIND IN THE VEHICLE.” After this image notification has been sent to mobile computing device57, method200will move to completion202. In an alternative embodiment, if the search cannot find a sample image with an object like the one found in the test image302, server54can determine that the object detection technique has produced a false positive and method200will move straight to completion202. An example of a false positive would be when a tree leaf has somehow fallen onto the seat cushion (e.g., through an open window) sometime between the capturing of the reference and test image.

In step290, server54will generate a notification for the user and will collaborate with telematics unit24and/or mobile computing device57to implement this notification. In one embodiment, this notification is embodied as a visible text message displayed on the user interface59of the user's mobile computing device57or on an outward facing display located somewhere on the vehicle's12body. For example, this text message may state: “WARNING: CHECK REAR SEAT FOR MISLAID ITEMS” or the like. The test image302may also be sent to support this text message notification. In another embodiment, the notification is audible warning from a vehicle exterior located audio component64(e.g., outwardly facing speakers installed on the vehicle body) such as, for example, chirps or dings (e.g., via an electronic/software chime module) or an audio message stating: “WARNING: PLEASE CHECK REAR SEAT FOR MISLAID ITEMS.” In another embodiment, the notification is a temporary activation of the horn system (i.e., causing sequential light sounding horn honks). Skilled artists will see that server54may simultaneously generate and implement multiple notification embodiments (e.g., implementing both the text and audio messages at the same time). Server54may also collaborate with the BCM44to unlock one or more vehicle doors13at the same time that the notification is being provided (this may be regardless of whether a vehicle-share reservation has come to completion).

In addition, when an object recognition technique is performed and the object in the test image is recognized as a certain type, in step290, server54may take additional precautionary steps to protect the object(s) found in the backseat. For example, if the object recognition technique recognizes the object left behind as a helpless animal, child, or infant, server54may remotely command the BCM44to cause the door13to be unlocked and/or released from the door fame (i.e., by sequentially unlocking, unlatching, and rotating away or sliding open the door13). Alternatively, or simultaneously, server54may also remotely command BCM44to open one or more power windows of the vehicle12in an effort to ventilate the vehicle interior. Server54may additionally or alternatively contact the emergency services provider75that there is a helpless animal, child, or infant trapped in vehicle12. As follows, server54may collaborate with GPS chipset/component42to get the location of vehicle12and also provide that location to the emergency services provider75. After step280, method200moves to completion202. It has also been envisioned, that the object in the test image is recognized as an object commonly left in the back seat of the vehicle12(e.g., a baseball bat and ball often left in the backseat) or the type of object generally known to be left in vehicles (e.g., a pack of gum or magazines), then server54will refrain from generating a notification for the user and method200will move from optional step280to completion202.

None of the elements recited in the claims are intended to be a means-plus-function element within the meaning of 35 U.S.C. § 112(f) unless an element is expressly recited using the phrase “means for,”