Patent Description:
The invention relates to the field of known autonomous driver-assistance systems (ADAS) as used for autonomously driving a vehicle. Such advanced driver-assistance systems in vehicles may include Valet Parking Assistance (VaPA) to provide fully automated steering and manoeuvring when parking, for example within a car park or parking structure. Such systems use automated vehicle controls such as GPS (Global Positioning System) or on-board sensors along with camera, lidar, radar proximity and ultrasonic sensors, to navigate, identify valid parking slots, and park the vehicle ("drop-off" manoeuvre). The vehicle is also able to autonomously drive the parked vehicle from a parking slot to a specified pickup location ("summon" manoeuvre) upon request by the user. Within a summon manoeuvre the vehicle drives along a specified route or distance. Consequently, the summon manoeuvre is an operation during which the vehicle drives (driving operation). The aforementioned "drop-off" manoeuvre more relates to a parking operation. Thus, in the present document the term "driving operation" is used synonymously with the term "drop-off manoeuvre" and the term "parking operation" is used" synonymously for the term "summon manoeuvre".

Autonomous driver-assistance systems (ADAS) require information about the area where they are applied to be mapped for the vehicle to plan a route for the drop-off and summon manoeuvre. This digital map of the area, for example a car park or parking structure, could be very simple and consist only of a description of the drivable sections, or more complex such as high-definition maps with additional attributes such as signs, lane widths and the like. In most cases the ADAS or VaPA has to consider an actual traffic situation in the area of use, for example the car park or parking structure. Said digital map and said actual traffic situation might permanently be updated by using dedicated databases being connected with the ADAS or the vehicle. Alternatively, or additionally said digital map and said actual traffic situation might be updated by GPS data or the use of on-board sensors along with camera, lidar, radar proximity and ultrasonic sensors. Also, data relating to said digital map or said actual traffic situation which might be tracked and shared by other traffic participants might be used for such an update.

When using ADAS or VaPA for the first time, a user might not be familiar with the functions of the system. The user might want to learn or check how the system works. One way of doing this is to allow the user to stay inside the vehicle during an automated driving or parking operation (e.g. "drop-off" manoeuvre or "summon" manoeuvre). However, at a certain point in time the driving or parking operation will have to be performed without the user being inside the vehicle. For this purpose it would be beneficial if the user could monitor the driving or parking operation and the respective vehicle behaviour during the driving or parking operation from outside the vehicle, in particular from a position where the vehicle is out of sight of the user.

Relevant prior art may be identified in <CIT>, <CIT> and <CIT>.

Document <CIT> refers to an apparatus including a plurality of capture devices and a processor. The plurality of capture devices may each be configured to generate video frames corresponding to an area outside of a vehicle. The processor may be configured to receive the video frames from each of the plurality of capture devices, generate video data for a display in response to the video frames, store a view preference for the display corresponding to (i) a location and (ii) a vehicle status, determine a current location and a current status of the vehicle and generate an output signal to select a view for the display. The output signal may be generated in response to the current location matching the location and the current status matching the vehicle status. The view selected may be determined based on the view preference.

<CIT> refers to a camera system installed on the front end of a vehicle, either on the left front, the right front, or both sides. The camera is linked via wired or wireless connection to an onboard computer and a navigation display that is located within the passenger compartment of the vehicle. The driver reviews a visual description on the display of any oncoming traffic in the form of motor vehicles, pedestrians, cyclists, animals and the like on the navigation display via a single screen, split screen or alternating screens. The camera system can include a speed sensor that detects when the vehicle reaches a threshold speed to activate or de-activate the camera. Alternatively, the computer can activate the system when a turn signal is activated, and de-activate the system when the turn signal is no longer activated. This camera system can be retrofitted into older vehicles.

<CIT> refers to a stereo camera-based autonomous driving method and apparatus, the method including estimating a driving situation of a vehicle, determining a parameter to control a stereo camera width of a stereo camera based on the estimated driving situation, controlling a capturer configured to control arrangement between two cameras of the stereo camera for a first direction based on the determined parameter, and measuring a depth of an object located in the first direction based on two images respectively captured by the two cameras with the controlled arrangement.

It is therefore an object of the invention to provide a user-friendly system and method that allows a user to monitor (and observe) an autonomous driving or parking operation of a vehicle from outside the vehicle. It is a further object of the invention to overcome or at least mitigate shortcomings of the prior art, or at least to provide a user with a convenient alternative.

These objects are achieved by the system and method defined in the appended claims.

According to a first aspect of the invention, a system for monitoring an autonomous driving or parking operation of a vehicle is provided according to claim <NUM>.

The portable electronic device may be a any portable computer, e.g. a laptop, notebook, tablet computer, telephone, smartphone or the like. In principle also a stationary computer could be employed instead of a portable electronic device as the basic idea of the present invention relates to remotely observing a driving or parking operation of the vehicle.

The system allows a user of a portable electronic device (the second communication unit of which is wirelessly connected to a first communication unit of the vehicle) to visually observe an autonomous driving or parking operation of the vehicle in a live-mode or live-video (during the autonomous driving or parking operation) from a position external to the vehicle. The cameras installed at different positions of the vehicle may be installed outside or inside the vehicle. Each camera may include one or more lenses. Also, each camera may be operated by a microcontroller, the microcontroller being connected with a main control unit of the vehicle. The mentioned expression of "capturing" videos may be understood in terms of "displaying" moving images (time-resolved image-sequences captured by a camera) to a user. Otherwise the term "capturing" may be understood in terms of recording (and storing) said moving images (time-resolved image-sequences captured by a camera). Data corresponding to said moving images may be stored temporarily or for longer terms. Said data may also be transmitted to an external server or database (e.g. a cloud). The signal connections of the cameras and the first communication unit may be based on cable(s) or may be a wireless signal connection. The first communication unit may be part of a main control unit of the vehicle. The wireless signal connection between the first and second communication unit may be based on digital signal (or data) transmission. Said wireless signal connection may exemplarily be based on Bluetooth, WLAN, ZigBee, NFC, Wibree, WiMAX, IrDA, FSO, LiFi. Said wireless signal connection may also be based on mobile internet connections of the first and second communication units, e.g. mobile internet connections based on <NUM>, <NUM>, <NUM>, <NUM> or any other known or future standard for mobile internet connections. The wireless connection between said first and second communication unit may be a direct connection (including a direct signal and data transfer) between both units, or may be an indirect connection including one or more intermediate transmission units or server. The first and second communication units can be understood as communication interfaces, each comprising dedicated means (e.g. antennas) for receiving and transmitting signals and data.

A suitable application software (abbrev. : App) may be installed on the portable electronic device to operate a visualization of the data transmitted to the portable electronic device via the wireless connection of the first and second communication unit. The App may be configured to overlay or display specific features/information directly in the video or next to the video.

As mentioned above, the portable electronic device is configured to show said live-videos to a user of the portable electronic device in a live mode during the driving or parking operation. This enables a user to observe an actual autonomous driving or parking operation via his smartphone, although the user may be located out of sight of the vehicle. The user may thus observe the vehicle behaviour and its vicinity in real time and on demand.

The view-management means may comprise hardware and software components, both being part of the portable electronic device or the vehicle. It is also possible, that hardware and software components of the vehicle and the portable electronic device define the view-management means and are configured to interact with each other. Hardware components may be understood as computing unit. Besides the possibility of automatically selecting a camera-view by the view-management means, the latter may be configured in that a user can manually switch between different camera-views. The view-management means may comprise an algorithm (the algorithm may be based on artificial intelligence) that may be operated in a dedicated software (environment), the software being installed on one or both of said computing units. The automated selection of the camera-view of which the corresponding live-video is shown to the user is performed by said algorithm.

When referring to said automated selection of the camera-view, the view-management means provide a situation-based view management system. Based on the situational context, a camera-view may be changed automatically to show the most interesting/relevant camera-view to the user. The algorithm may consider different criteria when calculating which camera-view (of which camera) is to be shown to the user. Said criteria may relate to the autonomous driving or parking operation as such, to the vicinity of the vehicle (e.g. the traffic situation, traffic participants) or to the needs of the user.

According to another aspect of the invention, the cameras are installed at positions of the vehicle to provide the following camera-views: a front view of the vehicle, a rear view of the vehicle, a left view of the vehicle and a right view of the vehicle. To provide each of said views (front view, rear view, left view, right view) a single camera or a number of cameras may be provided at the relevant positions of the vehicle (the front, the back, the left, the right of the vehicle). The cameras may be mounted on suitable vehicle components. It is to be noted that the system according to the invention may be implemented in newly fabricated vehicles or via retrofitting.

According to another aspect of the invention, a live-video referring to a bird's eye view of the vehicle can be obtained based on the live videos provided by the cameras installed at the vehicle and/or position data of the vehicle. A bird's eye view refers to a view of the vehicle from above, with a perspective as the observer were a bird. The live-video in bird's eye view may be calculated (extrapolated) based on video data provided from the front, rear, left, and/or right camera of the vehicle. Additionally, or alternatively one or more camera(s) may be installed on top of the roof of the vehicle. Said camera (being installed on the roof) may be a <NUM>° camera. It could also be possible to install a drone at the vehicle. In case a bird's eye view would be needed, the drone could rise (fly) to a certain height above the vehicle and provide a bird's eye view.

According to another aspect of the invention, the view-management means are configured to select one- or more camera-views of which the corresponding live-video(s) is/are shown to the user on the portable electronic device. It is important to note that the view-management means are not only suitable to select a single camera view, but also to select multiple camera views to be shown to a user at the same time. In many driving or parking operations (as well as traffic situations) a parallel observation of several (different) views may be of interest. The live-videos may be shown to the user in a gallery format with multiple videos displayed to the user. The gallery format may include the videos as video-mosaics.

According to another aspect of the invention, the view-management means are configured to select a camera-view of which the corresponding live-video is shown to the user on the portable electronic device in a single camera-view or as highlighted camera-view besides other views. Sometimes it might be of advantage to only display a single camera-view to the user, e.g. in situations where the other accessible camera-views would not provide sufficient information referring to an actual driving or parking operation (or to an actual traffic situation). A single camera-view is means that only a single live-video (referring to a specific) camera-view is displayed to the user. A highlighted camera-view is to be understood as display mode where a live-video referring to a specific camera-view is prominently displayed to a user besides live-videos of other camera-views (which are not highlighted). The live video referring to the highlighted camera-view is shown enlarged with respect to live-videos of other camera-views shown to the user.

According to another aspect of the invention, the view-management means are configured to automatically select the single or highlighted camera-view as follows:.

The view-management means may be configured to evaluate (or weigh) which of the situations/aspects given under lit. - d is most relevant at a certain point in time. According to the evaluation (weighing) it is then decided which of the camera-views is selected as single or highlighted camera-view.

When referring to lit. it is to be mentioned, that a camera directed to the direction of movement (straight ahead, straight reverse) may be defined as the most relevant camera. When a change in the direction of movement occurs (lit. b), the camera-view may change to a camera-view of a camera directed in the changed direction of movement. A driving tube view may be overlaid on top of the selected camera view to indicate the direction of movement.

The case of lit c. is directed to a situation where an object is present within a predefined first distance (or range) around the vehicle. The object might be a pedestrian. The system may comprise means or determining the distance between the object and the vehicle. Also, the system may comprise means for determining if the vehicle is moving toward the object (e.g. the distance between object and vehicle decreases). Said means may be one of the cameras as such or additional means (distance measurement means) installed at the vehicle. If both criteria are met, a camera-view directed to the object is selected (shown as single camera-view or highlighted with respect to other camera-views). Said camera-view may be called "static object view". Said predefined first distance may automatically be determined or may be continuously adapted to a situational context (e.g. the traffic situation) of the vehicle.

where an object is monitored within a predefined second distance from the vehicle (the second predefined distance being larger than the first predefined distance) and a condition where the object moves toward the vehicle (distance between object and vehicle is reduced, which can be detected by the camera as such or dedicated distance measurement means), a camera-view is selected which is directed to the object as single or highlighted camera-view. Said predefined second distance may automatically be determined or may be continuously adapted to a situational context (e.g. the traffic situation) of the vehicle.

According to another aspect of the invention, in case d. , the single or highlighted camera-view may be switched when the object leaves a field of view of a first camera and enters a field of view of a second camera. As the field of view of the cameras may be restricted, a moving object might enter different field of views of different cameras. To visually follow the movement of the object, it might be necessary to switch the selected camera according to the field of view in which the moving object is actually present. The live-video may be provided with a bounding box to indicate which moving (dynamic) object is actually tracked. The bounding box may be bound to the moving object and may be provided as overlay of the live-video.

According to another aspect of the invention, the view-management means are configured to consider an anticipated length of movement of the autonomous driving or parking operation for automatically selecting the single or highlighted camera-view shown to the user, wherein in case that an anticipated length of movement is below a given threshold length, the selected camera view(s) shown to the user are fixed. Said feature avoids flickering of camera-views due to fast changes of the direction of movement (e.g. within parking operations when the vehicle needs to undergo short moves). The length of movement may be anticipated based on path planning information (e.g. GPS data) or a tracked vehicle behaviour. In case of vehicle movements below a given threshold length, the camera-view should not be changed. The given threshold length may be automatically determined or manually defined by a manufacturer of the vehicle, a user of the vehicle or the like.

According to another aspect of the invention, the view-management means are configured to select the bird's view as single or highlighted camera-view in case that multiple movements of the vehicle with an anticipated length of movement below said given threshold value are expected. A bird's eye view does not require fast changes of camera-views, much more the autonomous driving or parking operation (including multiple changes in the direction of movement) may be observed from a position above the vehicle. To help the user to understand the vehicle behaviour (the short movements), an anticipated final position of the vehicle and the intended path (the driving or parking operation relies on) may be projected on top of the camera-view. Such a camera-view may be called "parking view". So one further aspect of the invention enables that an anticipated movement path or end position of the vehicle in the autonomous driving or parking operation is projected into the single or highlighted camera-view.

According to another aspect of the invention, the view-management means are configured to automatically select the single or highlighted camera-view in predefined situations of an autonomous driving or parking operation according to predefined selection criteria, wherein the predefined situations and predefined selection criteria are as follows:.

When referring to case a. , a parking operation might often be better observed from a bird's eye view. When referring to case b. , an automated switching through accessible or predefined camera-views (e.g. front, rear, left, right) at the beginning (or before) or at the end of an autonomous driving or parking operation enables a user to observe or check the vicinity of the vehicle for possible objects that could hinder the driving or parking operation.

To select the camera-view according to the above given criteria a. , the system may consider path planning data (e.g. based on GPS data) or data referring to the local environment of the vehicle. Path planning data may also refer to a local map. Such data may be provided from an external server to the vehicle or the portable electronic device, so that the view-management means may consider said data.

Optionally, a section of the live-video (of a certain camera-view) where the vehicle is assumed to get very close to a certain object during the driving or parking operation may be marked (e.g. with a bounding box).

As mentioned before, the view-management means are configured to automatically select the camera-view shown to the user based on a routine, optionally a routine based on artificial intelligence.

According to another aspect of the invention, the view management means are configured to show additional information to a user by displaying said information in the live-video corresponding to a selected camera-view shown to the user on the portable electronic device, wherein said information is/are preferably displayed as video-overlay(s). Said additional information may also be displayed by boxes or illustrative means affixed to objects or positions present in the live-video. The information may relate to anticipated movement paths, vehicle data, data referring to the environment (e.g. an outdoor temperature), traffic signs etc..

Additional situations where an overlay of the live-video would help the user to better understand the situational context and the behaviour of the vehicle include (but are not limited to):.

The system may be configured to include said overlays to the live-video(s) displayed to a user on a screen of the portable electronic device.

Additionally, a function may be implemented in the system where the user may choose to shut off the automated view-selection and to select a camera-view manually. This feature may be implemented in the App operated on the portable electronic device. The user may also choose a hybrid mode where some camera-views may be fixed (as selected by the user) and other views change automatically according to the situational context.

According to yet another aspect of the invention, there is provided a method according to claim <NUM> for monitoring an autonomous driving or parking operation of a vehicle with a previously described system.

One or more camera-views of which the corresponding live-video is/are shown to a user on the portable electronic device in a single camera-view or as highlighted camera-view besides other views is/are automatically selected by view-management means. The selection may refer to a selection (and display) of a single camera-view or to a selection of a highlighted camera-view (a selected camera-view is displayed enlarged with respect to other camera-views). The automated selection may be based on the same criteria or situations as described before.

It is to be emphasized, that the system may comprise dedicated units or means for performing any of the method steps described above.

The invention will now be described in more detail with reference to the appended figures. In the figures:.

Turning to <FIG>, a vehicle to be used in a system according to the invention is shown in a schematic illustration. The system is suitable for monitoring an autonomous driving or parking operation of the vehicle <NUM>. The vehicle <NUM> (e.g. a car) has a front F, a back B as well as a left side L and right side R. A number of cameras <NUM>, 2R, 2F, 2B are installed at different positions of the vehicle. A camera <NUM> is installed at the left side L of the vehicle <NUM>, a camera 2R is installed at the right side R of the vehicle <NUM>, a camera 2F is installed at the front F of the vehicle <NUM>, and a camera 2B is installed at the back side B of the vehicle <NUM>. The back B may synonymously be expressed as "rear" side of the vehicle <NUM>. The positions of the cameras <NUM>, 2R, 2F, 2B were only chosen for illustrative purposes. Each of the cameras <NUM>, 2R, 2F, 2B is configured to capture live-videos of the driving or parking operation from a camera-view corresponding to the position of the camera <NUM>, 2R, 2F, 2B. The corresponding camera-views are indicated with field-of-views <NUM>, <NUM>, <NUM> and <NUM>, wherein the field-of-view <NUM> refers to camera <NUM>, field-of-view <NUM> refers to camera 2R, field-of-view <NUM> refers to camera 2F and field-of-view <NUM> refers to camera 2B. The vehicle comprises a first communication unit <NUM> which may be part of a board computer of the vehicle <NUM>. The cameras <NUM>, 2R, 2F, 2B are in signal connection with the first communication unit <NUM>.

As shown in <FIG>, the system according to the invention comprises a number of cameras <NUM>, 2R, 2F, 2B installed at different positions of the vehicle <NUM>, each of the cameras <NUM>, 2R, 2F, 2B configured to capture live-videos of the driving or parking operation from a camera-view corresponding to the position of the camera <NUM>, 2R, 2F, 2B, wherein the cameras <NUM>, 2R, 2F, 2B are in signal connection (not shown) with a first communication unit <NUM> being installed in the vehicle <NUM>.

The system further comprises a portable electronic device <NUM> comprising a second communication unit <NUM>. The portable electronic device <NUM> comprises a display <NUM>. The portable electronic device <NUM> is used by user <NUM>, wherein the user <NUM> is located external to the vehicle <NUM>.

The first communication unit <NUM> is configured to transmit the captured live-videos to the second communication unit <NUM> via a wireless signal connection <NUM>, wherein the second communication unit <NUM> is configured to receive the transmitted live-videos, and wherein the portable electronic device <NUM> is configured to show the live-videos to the user <NUM> of the portable electronic device <NUM> in a live mode during the driving or parking operation. The system further comprises view-management means (not shown) configured to automatically select a camera-view of which the corresponding live-video is shown to the user <NUM> on the portable electronic device <NUM>. The view-management means may comprise hardware and software components, both being part of the portable electronic device <NUM> or the vehicle <NUM>. It is also possible, that hardware and software components of the vehicle <NUM> and the portable electronic device <NUM> together provide the view-management means and are configured to interact with each other. Hardware components may be understood as computing unit. Besides the possibility of automatically selecting a camera-view by the view-management means, the latter may be configured in that a user <NUM> can manually switch between different camera-views. The view-management means may comprise an algorithm (the algorithm may be based on artificial intelligence) that may be operated in a dedicated software (environment), the software being installed on one or both of said computing units. The automated selection of the camera-view of which the corresponding live-video is shown to the user <NUM> is performed by said algorithm.

As shown in <FIG>, the view-management means are configured to select a camera-view of which the corresponding live-video is shown to the user <NUM> on the portable electronic device <NUM> in a single camera-view <NUM> or as highlighted camera-view <NUM> besides other views <NUM>. In a single camera-view <NUM> only a single live-video is displayed on the display <NUM> of the portable electronic device <NUM>. In a highlighted camera-view <NUM> a live-video of a certain camera view is displayed enlarged when compare to the live-videos of other views <NUM> (shown smaller).

<FIG> illustrate different buttons (provided in an App operated on the portable electronic device <NUM>) which a user <NUM> of the portable electronic device <NUM> may activate/deactivate, wherein the buttons are related to different selection options referring to the selection of a camera-view of which a live-video is shown to the user <NUM>. The buttons may be shown in a touch sensitive manner on the display <NUM> of the portable electronic device <NUM>. From the right to the left of the buttons illustrated in <FIG> the buttons refer to a right view, a left view, a rear view, a front view and a bird's eye view of the vehicle <NUM>. Said buttons may also be displayed in an on-board display of the vehicle <NUM>, so that the user <NUM> of the vehicle may pre-select a certain selection procedure before leaving the vehicle <NUM>.

Claim 1:
System for monitoring an autonomous driving or parking operation of a vehicle (<NUM>), the system comprising
- a number of cameras (<NUM>, 2R, 2F, 2B) configured to be installed at different positions of the vehicle (<NUM>), each of the cameras (<NUM>, 2R, 2F, 2B) configured to capture live-videos of the driving or parking operation from a camera-view corresponding to the position of the camera (<NUM>, 2R, 2F, 2B), wherein the cameras (<NUM>, 2R, 2F, 2B) are in signal connection with a first communication unit (<NUM>) configured to be installed in the vehicle (<NUM>);
- a portable electronic device (<NUM>) comprising a second communication unit (<NUM>);
wherein the first communication unit (<NUM>) is configured to transmit the captured live-videos to the second communication unit (<NUM>) via a wireless signal connection (<NUM>), wherein the second communication unit (<NUM>) is configured to receive the transmitted live-videos, and wherein the portable electronic device (<NUM>) is configured to show the live-videos to a user (<NUM>)of the portable electronic device (<NUM>) in a live mode during the driving or parking operation,
wherein the system comprises view-management means configured to automatically select a camera-view of which the corresponding live-video is shown to the user (<NUM>) on the portable electronic device (<NUM>) in a single camera-view (<NUM>) or as highlighted camera-view (<NUM>) besides other views (<NUM>), wherein in a highlighted camera-view (<NUM>) a live-video of the selected camera-view is displayed enlarged as compared to the live-videos of other camera-views,
wherein the cameras (<NUM>, 2R, 2F, 2B) are configured to be installed at positions of the vehicle (<NUM>) to provide the following camera-views: a front view of the vehicle (<NUM>), a rear view of the vehicle (<NUM>), a left view of the vehicle (<NUM>) and a right view of the vehicle (<NUM>),
wherein the view-management means are configured to select a bird's eye view as the single (<NUM>) or highlighted (<NUM>) camera-view in case that multiple movements of the vehicle (<NUM>) with an anticipated length of each of said multiple movements below a given threshold value are expected,
said multiple movements including multiple changes in the direction of movement.