Vehicle control systems and methods and optical tethering techniques

The present disclosure generally relates to vehicle remote parking assistance, and, more specifically, systems and methods for determining a distance between a vehicle and a mobile device during a remote parking procedure. In particular, the systems and methods include selecting a camera lens or focal length for use with a first method of determining a distance between a vehicle and a mobile device based on an image of the vehicle and a digital model of the vehicle.

BACKGROUND

Vehicles may include the ability to remotely drive themselves with no or only minor control from a user. Some vehicles may be able to park themselves while an owner or driver watches from outside the vehicle and provides no or minimal motion control. In these instances, the driver may initiate the remote parking operation from a mobile device in communication with the vehicle. The mobile device may thus be able to control one or more aspects of the vehicle remotely.

In these cases of remote control of the vehicle, the user may be required to remain within a certain distance of the vehicle while controlling the vehicle. Some states, countries, or other jurisdictions may require that a driver be within a threshold distance from the vehicle in order to allow this remote parking feature to take place. For instance, a regulation or law may require a driver to be within 10 meters of the vehicle during operation of the remote parking feature and may require that the vehicle stop if the driver is too far away.

DETAILED DESCRIPTION

Overview

The present disclosure generally relates to vehicle remote parking assistance and, more specifically, systems and methods for determining a distance between a vehicle and a mobile device during a remote parking procedure by selecting a lens and focal length of the mobile device and zoom configuration for optical tethering between the mobile device and the vehicle. In particular, the systems and methods include selecting a camera lens and/or focal length for use with a first method of determining a distance between a vehicle and a mobile device based on an image of the vehicle and a digital model of the vehicle.

An exemplary vehicle control system110includes a vehicle100including a vehicle computer112and marker features114. The vehicle control system110also includes a mobile device118including at least one camera120(and in some instances a plurality of cameras). The camera120includes at least one lens (and in some instances a plurality of lenses) and a plurality of focal lengths.

Focal length relates to the angle of view (or field of view), which is how much of a scene will be captured, and the magnification, which is how large individual elements will be. The longer the focal length, the narrower the angle of view and the higher the magnification. The shorter the focal length, the wider the angle of view and the lower the magnification.

For purposes of illustration, the camera120includes a first lens122with a first focal length (represented by a first field of view124) and a second lens126with a second focal length (represented by a second field of view128) where the first focal length is longer than the second focal length. In some cases, the camera120may include a zoom lens with a variable focus length.

The vehicle control system110is configured to determine a distance130between the vehicle100and the mobile device118during a remote parking procedure or other Remote Driver Assist Technology (ReDAT) operation.

A first method of determining the distance130is based on an image240of the vehicle100from the camera120of the mobile device118and a 3D model242of the vehicle100. The vehicle100may transmit the 3D model242of the vehicle100to the mobile device118. The mobile device118is configured to receive the 3D model242, capture an image240of the vehicle100with the camera120, and determine a distance130between the mobile device118and vehicle100based on the image240and the 3D model242.

In particular, the mobile device118may overlay a graphic244over the image240on a display246of the mobile device118. The overlayed graphic244designates where to position an image of the vehicle100on within the image240or display246when performing the first method of determining the distance130. For example, the first method is configured to use a portion of the image240that is within a boundary defined by the graphic244.

The mobile device118may also display a touchpoint248on the display246. For example, the user must maintain contact with the touchpoint248to use the vehicle control system110.

A second method of determining the distance130can include using an image from a camera of the vehicle100(e.g., imaging techniques to detect face or posture of a user, or to detect the mobile device118), a strength of a wireless connection between the vehicle100and the mobile device118(e.g., Bluetooth or Wi-Fi), low frequency (LF) antennas to triangulate the position of a LF key fob, determining the distance based on a calculated trajectory of the vehicle100and a calculated trajectory of the mobile device118, combinations thereof, and the like.

According to an exemplary method, the mobile device may select a lens122,126, based on the distance130, for use with the first method of determining the distance130. The distance130for selecting a lens122,126may be determined by the first method or a second method. For example, if the distance130is greater than a focal length threshold distance150, the first lens122with the longer focal length and the narrower field of view124is selected. Alternatively, if the distance130is less than the focal length threshold distance150, the second lens126with the shorter focal length and the wider field of view128is selected.

The focal length threshold distance150may be selected such that a portion of the vehicle100(including the entire vehicle100) that facilitates performing the first method (e.g., identifying marker features114in the image240) is within the field of view124,128. The focal length threshold distance150may also be selected such that a the portion of the vehicle100that facilitates performing the first method (e.g., identifying marker features114in the image240) is shown with sufficient magnification.

If the mobile device118is unable to perform the first method with the selected lens (e.g., first lens122) may switch to a wider lens (e.g., second lens126) and/or provide instructions to the user via a display of the mobile device118such that the image of the vehicle100is positioned in the overlay graphic244.

If the distance130is not known or is subject to an error calculation such that the distance130may not be reliably compared to the focal length threshold distance150, the second lens126with the shorter focal length and the wider field of view128may be initially selected as a default.

The distance130may be compared to a threshold operational distance160(e.g., 6 meters) that determines if the vehicle control system110can be used to control the vehicle100. For example, if the calculated distance130is greater than the threshold operational distance160, control with the mobile device118is disabled until the distance130is less than the threshold operational distance160.

The mobile device118may display the graphic244with a color (or other indicator including visual, audible, and tactile indicators) based a status when performing the first method of determining the distance130. For example, if a status is that the distance130is not determined by the first method, the color of the graphic244may be red. If a status is that the distance130is determined by the first method, the color of the graphic244may be green.

If the distance130nears the threshold operational distance160(e.g., distance130is greater than a warning distance162), the color of the graphic244may be yellow. In addition, the mobile device118may provide additional notices including text, audible warnings or instructions, additional graphics overlayed on the image240such as an arrow270, vibration or other haptic signals, zoom sequences, combinations thereof, and the like. Similarly, the mobile device118may provide these or similar notices if the distance130exceeds the threshold operational distance160.

These and other advantages of the present disclosure are provided in greater detail herein.

ILLUSTRATIVE EMBODIMENTS

The vehicle control system110includes the vehicle100and the mobile device118. The vehicle100includes the vehicle computer112and marker features114. The vehicle control system110is described in further detail with respect toFIG.4.

The vehicle100may take the form of another passenger or commercial automobile such as, for example, a truck, a car, a sport utility vehicle, a crossover vehicle, a van, a minivan, a taxi, a bus, etc., and may be configured to include various types of automotive drive systems. Example drive systems can include various types of internal combustion engine (ICE) powertrains having a gasoline, diesel, or natural gas-powered combustion engine with conventional drive components such as, a transmission, a drive shaft, a differential, etc.

In another configuration, the vehicle100may be configured as an electric vehicle (EV). More particularly, the vehicle100may include a battery EV (BEV) drive system. The vehicle100may be configured as a hybrid EV (HEV) having an independent onboard power plant or a plug-in HEV (PHEV) that includes a HEV powertrain connectable to an external power source (including a parallel or series hybrid powertrain having a combustion engine power plant and one or more EV drive systems). HEVs can include battery and/or super capacitor banks for power storage, flywheel power storage systems, or other power generation and storage infrastructure.

The vehicle100may be further configured as a fuel cell vehicle (FCV) that converts liquid or solid fuel to usable power using a fuel cell, (e.g., a hydrogen fuel cell vehicle (HFCV) powertrain, etc.) and/or any combination of these drive systems and components.

Further, the vehicle100may be a manually driven vehicle, and/or be configured to operate in a fully autonomous (e.g., driverless) mode (e.g., level-5 autonomy) or in one or more partial autonomy modes. Examples of partial autonomy modes are widely understood in the art as autonomy Levels 1 through 5.

An autonomous vehicle (AV) having Level 1 autonomy may generally include a single automated driver assistance feature, such as steering or acceleration assistance. Adaptive cruise control is one such example of a Level-1 autonomous system that includes aspects of both acceleration and steering.

Level-2 autonomy in vehicles may provide partial automation of steering and acceleration functionality, where the automated system(s) are supervised by a human driver that performs non-automated operations such as braking and other controls.

Level-3 autonomy in a vehicle can generally provide conditional automation and control of driving features. For example, Level-3 vehicle autonomy typically includes “environmental detection” capabilities, where the vehicle can make informed decisions independently from a present driver, such as accelerating past a slow-moving vehicle, while the present driver remains ready to retake control of the vehicle if the system is unable to execute the task.

Level 4 autonomy includes vehicles having high levels of autonomy that can operate independently from a human driver, but still include human controls for override operation. Level-4 automation may also enable a self-driving mode to intervene responsive to a predefined conditional trigger, such as a road hazard or a system failure.

Level 5 autonomy is associated with autonomous vehicle systems that require no human input for operation, and generally do not include human operational driving controls.

The mobile device118includes a mobile device computer210(e.g., as described with respect to vehicle computer112), the touchscreen display246, and the camera120. The mobile device computer210includes a memory220and a processor230. The memory220include the digital model242and computer executable instruction that, when execute by the processor230, cause the processor to perform methods described herein.

Broadly, the mobile device118is configured to interact with the vehicle100.

The camera120includes at least one lens and a plurality of focal lengths. Focal length relates to the angle of view (or field of view), which is how much of a scene will be captured, and the magnification, which is how large individual elements will be. The longer the focal length, the narrower the angle of view and the higher the magnification. The shorter the focal length, the wider the angle of view and the lower the magnification.

For purposes of illustration, the camera120includes the first lens122with the first focal length (represented by the first field of view124) and the second lens126with the second focal length (represented by the second field of view128) where the first focal length is longer than the second focal length. In some cases, the camera120may include a zoom lens with variable focus lengths.

The mobile device118may overlay the graphic244over the image240on the display246of the mobile device118. The overlayed graphic244designates where to position the vehicle100within the image240when performing the first method of determining the distance130. For example, first method is configured to use a portion of the image240that is within a boundary defined by the graphic244.

The mobile device118may also display a touchpoint248on the display246. For example, the user must maintain contact with the touchpoint248to use the vehicle control system110.

The vehicle control system110is configured to determine a distance130between the vehicle100and the mobile device118during a remote parking procedure. The distance130may be compared to the threshold operational distance160(e.g., 6 meters). The threshold operational distance160is a maximum distance at which the vehicle control system110can be used to control the vehicle100. For example, if the distance130is greater than the threshold operational distance160, control of the vehicle100with the mobile device118is disabled by the vehicle control system110until the distance130is less than the threshold operational distance160.

The vehicle control system110is configured to determine a distance130between the vehicle100and the mobile device118during a remote parking procedure.

A first method of determining the distance130is based on an image240of the vehicle100from the camera120of the mobile device118and the digital model242of the vehicle100. The vehicle100may transmit the digital model242of the vehicle100to the mobile device118. The mobile device118is configured to receive the digital model242, capture an image240of the vehicle100, and determine a distance130between the mobile device118and vehicle100based on the image240and the digital model242.

In particular, the vehicle100may include marker features114(e.g., lights) at marker locations. Each marker location on the vehicle100is associated with a coordinate of a vehicle coordinate system (e.g., of the digital model242) such that the marker locations, and distances and spatial relationships between the marker features114, are known.

Each marker feature114includes a unique pattern and/or other marker characteristics that are configured to be detected by image processing, pattern recognition, and/or other computer vision techniques. The marker feature114can be read to access associated marker information such as a name of the marker feature114, the marker location, and spatial relationships (e.g., distances to, directions to) with other marker features114.

The camera120is configured to capture the camera image240(e.g., image data) of the vehicle100and the marker features114. The mobile device118is configured to determine an image location of each marker feature114in the camera image240on a screen coordinate system.

The mobile device118includes a computer vision application that is configured to perform image processing, pattern recognition, and/or other computer vision techniques to read the marker features114in the camera image240and obtain the marker locations of each marker feature114from the associated marker information.

The mobile device118determines a transformation matrix (e.g., a camera model) that relates the image location of a marker feature114in the camera image240and the marker location of the marker feature114(e.g., of the digital model242). The transformation matrix reflects the pose of the camera120including the location and the orientation of the camera120.

The orientation of the camera120can be expressed as a rotation matrix (e.g., as rotation angles). The location of the camera120can be expressed by a translation vector. The transformation matrix may also include camera-specific adjustments such as for focal length, image sensor orientation, and size.

As the image locations of marker features114are known (e.g., determined from the camera image240) and the marker locations of the marker features114are known (e.g., the marker location is accessed from the marker information), the elements of the transformation matrix may be determined, for example, using an iterative optimization method. A pose of the camera120, including the orientation and location of the camera120, can be determined from the elements of the transformation matrix.

The distance130can be determined from the location of the camera120.

A second method of determining the distance130may be based on using an image from a camera of the vehicle100(e.g., imaging techniques to detect face, posture, the mobile device118), a strength of a wireless connection between the vehicle100and the mobile device118(e.g., Bluetooth or Wi-Fi), low frequency (LF) antennas to triangulate the position of a LF key fob or the mobile device118, based on a calculated trajectory of the vehicle100and a calculated trajectory of the mobile device118, combinations thereof, and the like.

An error in the distance calculation accumulates over time and is also calculated. If the error calculation becomes greater than a threshold error, another method of measuring distance may need to be used. The method that is used may be selected based on lowest amount of error.

Referring toFIG.3, the vehicle control system110may perform an exemplary method300. According to a first step310, the vehicle control system110may determine if the distance130between the vehicle100and the mobile device118has been previously determined. For example, the distance130may have been previously determined according to the first method or a second method described above.

According to a second step320, if the distance130between the vehicle100and the mobile device118has been previously determined, the mobile device118selects one of a plurality of focal lengths of the camera120(e.g., one of the lenses122,126) of the mobile device118based on the distance130.

For example, if the previously determined distance130is greater than the focal length threshold distance150, the first lens122with the longer focal length and the narrower field of view124is selected. Alternatively, if the distance130is less than the focal length threshold distance150, the second lens126with the shorter focal length and the wider field of view128is selected.

The focal length threshold distance150may be set such that a portion of the vehicle100(including the entire vehicle100) that facilitates performing the first method (e.g., identifying marker features114in the image240) is within the field of view124,128and/or has sufficient magnification.

According to a third step330, the mobile device118performs the first method to determine if a current distance130can be determined based on an image240of the vehicle100from the camera120using the selected focal length (e.g., lens) and the digital model242of the vehicle100.

According to a fourth step340, if the mobile device118is unable to determine the distance130according to the first method with the selected lens (e.g., first lens122), the mobile device118may switch to a wider lens (e.g., second lens126) and/or provide instructions to the user via a display of the mobile device118such that the image of the vehicle100is positioned in the overlay graphic244. The method300may return to step330after step340.

Following the first step310, according to a fifth step350, if the distance130is not previously determined or is subject to an error calculation (e.g., too much time has elapsed since the distance was previously determined or the distance was determined according to a method with a large amount of error) such that the distance130may not be reliably compared to the focal length threshold distance150, the second lens126with the shorter focal length and the wider field of view128may be initially selected as a default. The method300may proceed to step330from step350.

The mobile device118may display the graphic244with a color (or other indicator including visual, audible, and tactile indicators) based a status when performing the first method of determining the distance130. For example, if a status is that the distance130is not determined by the first method (e.g., at fourth step340or fifth step350), the color of the graphic244may be red.

According to a sixth step360, the vehicle control system110determines if the distance130is greater than the threshold operational distance160(e.g., 6 meters) to determine if the vehicle control system110can be used to control the vehicle100.

According to a seventh step370, if the distance130is greater than the threshold operational distance160, control with the mobile device118is disabled until the distance130is less than the threshold operational distance160. Here, the mobile device118may provide notices including text, audible warnings or instructions, additional graphics overlayed on the image240such as an arrow170, vibration or other haptic signals, zoom sequences, combinations thereof, and the like.

According to an eighth step380, if the distance130is less than the threshold operational distance160, the distance130is compared to the warning distance162. If the distance130is less than the warning distance162, the color of the graphic244may be green. If the distance130is greater than the warning distance162, the color of the graphic244may be yellow and/or the mobile device118may provide additional notices as described with respect to step370to attempt to prevent the distance130from exceeding the threshold operational distance160.

Referring toFIG.4, the vehicle control system110is described in greater detail.

The vehicle computer112includes computer components including a memory (e.g., memory400) and a processor (e.g., a processor402).

A processor may be any suitable processing device or set of processing devices such as, but not limited to: a microprocessor, a microcontroller-based platform, a suitable integrated circuit, one or more field programmable gate arrays (FPGAs), and/or one or more application-specific integrated circuits (ASICs).

Memory is computer readable media on which one or more sets of instructions, such as the software for performing the methods of the present disclosure, can be embedded. The instructions may embody one or more of the methods or logic as described herein. The instructions may reside completely, or at least partially, within any one or more of the memory, the computer readable medium, and/or within the processor during execution of the instructions.

The vehicle100includes a vehicle control unit (VCU)404, the VCU404includes a plurality of electronic control units (ECUs)410disposed in communication with the vehicle computer112. The VCU404may coordinate the data between vehicle systems, connected servers, and other vehicles operating as part of a vehicle fleet. The VCU404may control aspects of the vehicle100, and implement one or more instruction sets (e.g., method300) received from a vehicle system controller (such as vehicle computer112).

The VCU404can include or communicate with any combination of the ECUs410, such as, for example, a Body Control Module (BCM)412, an Engine Control Module (ECM)414, a Transmission Control Module (TCM)416, the Telematics Control Unit (TCU)418, a Restraint Control Module (RCM)420, and the like. The TCU418may be disposed in communication with the ECUs410by way of a Controller Area Network (CAN) bus440. In some aspects, the TCU418may retrieve data and send data as a CAN bus440node.

The CAN bus440may be configured as a multi-master serial bus standard for connecting two or more of the ECUs410as nodes using a message-based protocol that can be configured and/or programmed to allow the ECUs410to communicate with each other. The CAN bus440may be or include a high-speed CAN (which may have bit speeds up to 1 Mb/s on CAN, 5 Mb/s on CAN Flexible Data Rate (CAN FD)), and can include a low-speed or fault tolerant CAN (up to 125 Kbps), which may, in some configurations, use a linear bus configuration. In some aspects, the ECUs410may communicate with a host computer (e.g., the vehicle computer112, server(s), etc.), and may also communicate with one another without the necessity of a host computer.

The CAN bus440may connect the ECUs410with the vehicle computer112such that the vehicle computer112may retrieve information from, send information to, and otherwise interact with the ECUs410to perform steps described according to embodiments of the present disclosure. The CAN bus440may connect CAN bus nodes (e.g., the ECUs410) to each other through a two-wire bus, which may be a twisted pair having a nominal characteristic impedance. The CAN bus440may also be accomplished using other communication protocol solutions, such as Media Oriented Systems Transport (MOST) or Ethernet. In other aspects, the CAN bus440may be a wireless intra-vehicle CAN bus.

The VCU404may control various loads directly via the CAN bus440communication or implement such control in conjunction with the BCM412. The ECUs410described with respect to the VCU404are provided for exemplary purposes only, and are not intended to be limiting or exclusive. Control and/or communication with other control modules is possible, and such control is contemplated.

The ECUs410may control aspects of vehicle operation and communication using inputs from human drivers, inputs from a vehicle system controller, and/or via wireless signal inputs received via wireless channel(s) from other connected devices. The ECUs410, when configured as nodes in the CAN bus440, may each include a central processing unit (CPU), a CAN controller, and/or a transceiver.

The TCU418can be configured to provide vehicle connectivity to wireless computing systems onboard and offboard the vehicle100and is configurable for wireless communication between the vehicle100and other systems, computers, servers, devices, and modules.

For example, the TCU418includes a Navigation (NAV) system430for receiving and processing a GPS signal from a GPS432, a Bluetooth® Low-Energy Module (BLEM)434, a Wi-Fi transceiver, an Ultra-Wide Band (UWB) transceiver, and/or other wireless transceivers described in further detail below for using near field communication (NFC) protocols, Bluetooth® protocols, Wi-Fi, Ultra-Wide Band (UWB), and other possible data connection and sharing techniques.

The TCU418may include wireless transmission and communication hardware that may be disposed in communication with one or more transceivers associated with telecommunications towers (e.g., cellular towers) and other wireless telecommunications infrastructure. For example, the BLEM434may be configured and/or programmed to receive messages from, and transmit messages to, one or more cellular towers associated with a telecommunication provider, and/or and a Telematics Service Delivery Network (SDN) associated with the vehicle100for coordinating vehicle fleet.

The BLEM434may establish wireless communication using Bluetooth® and Bluetooth Low-Energy® communication protocols by broadcasting and/or listening for broadcasts of small advertising packets, and establishing connections with responsive devices that are configured according to embodiments described herein. For example, the BLEM434may include Generic Attribute Profile (GATT) device connectivity for client devices that respond to or initiate GATT commands and requests.

External servers (e.g., servers442) may be communicatively coupled with the vehicle100and the mobile device118via one or more network(s)452, which may communicate via one or more wireless channel(s)450. The wireless channel(s)450are depicted inFIG.4as communicating via the one or more network(s)452.

The mobile device118may be connected with the vehicle100via direct communication (e.g., channel454) using near field communication (NFC) protocols, Bluetooth® protocols, Wi-Fi, Ultra-Wide Band (UWB), and other possible data connection and sharing techniques.

The network(s)452illustrate example communication infrastructure in which the connected devices discussed in various embodiments of this disclosure may communicate. The network(s)452may be and/or include the Internet, a private network, public network or other configuration that operates using any one or more known communication protocols such as, for example, transmission control protocol/Internet protocol (TCP/IP), Bluetooth®, Wi-Fi based on the Institute of Electrical and Electronics Engineers (IEEE) standard 802.11, WiMAX (IEEE 802.16m), Ultra-Wide Band (UWB), and cellular technologies such as Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), High Speed Packet Access (HSPDA), Long-Term Evolution (LTE), Global System for Mobile Communications (GSM), and Fifth Generation (5G), Universal Mobile Telecommunications System (UMTS), Long Term Evolution (LTE), and the like.

The NAV system430may be configured and/or programmed to determine the vehicle location. The NAV system430may include a Global Positioning System (GPS) receiver configured or programmed to triangulate the vehicle location relative to satellites or terrestrial based transmitter towers associated with the GPS432.

The NAV system430may be further configured or programmed to develop routes from a current vehicle location to a selected destination, display a map and present directions to the selected destination, and determine an estimated time to travel to the selected location and a predicted time of arrival. The estimated time of arrival may be based on the position, speed, and heading or other vehicle information determined by the NAV system430.

The BCM412generally includes an integration of sensors, vehicle performance indicators, and variable reactors associated with vehicle systems, and may include processor-based power distribution circuitry that can control functions associated with the vehicle body such as lights, windows, security, door locks and access control, and various comfort controls. The BCM412may also operate as a gateway for bus and network interfaces to interact with remote ECUs.

The BCM412may coordinate any one or more functions from a wide range of vehicle functionality, including energy management systems, alarms, vehicle immobilizers, driver and rider access authorization systems, Phone-as-a-Key (PaaK) systems, driver assistance systems, Autonomous Vehicle (AV) control systems, power windows, doors, actuators, and other functionality, etc. The BCM412may be configured for vehicle energy management, exterior lighting control, wiper functionality, power window and door functionality, heating ventilation and air conditioning systems, and driver integration systems.

In other aspects, the BCM412may control auxiliary equipment functionality, and/or is responsible for integration of such functionality. In one aspect, a vehicle having a vehicle control system may integrate the system using, at least in part, the BCM412. For example, the BCM412may be used to control vehicle systems according to the determined distance130.

Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent upon reading the above description. The scope should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the application is capable of modification and variation. All terms used in the claims are intended to be given their ordinary meanings as understood by those knowledgeable in the technologies described herein unless an explicit indication to the contrary is made herein. In particular, use of the singular articles such as “a,” “the,” “said,” etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary. Conditional language, such as, among others, “can,” “could,” “might,” or “may,” unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments could include, while other embodiments may not include, certain features, elements, and/or steps. Thus, such conditional language is not generally intended to imply that features, elements, and/or steps are in any way required for one or more embodiments