Monitoring and planning a movement of a transportation device

The invention relates to a method, to a device, and to a computer-readable storage medium with instructions for monitoring the movement of a transportation device. In one embodiment, first, information about the trajectory of the transportation device is received by a mobile device. The trajectory is then displayed on a display unit of the mobile device in the form of an augmented reality representation. In response thereto, an input of the user of the mobile device for influencing the trajectory is detected. Finally, information is transmitted to the transportation device on the basis of the input of the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to German Patent Application No. DE 10 2018 215 186.5, filed on Sep. 6, 2018 with the German Patent and Trademark Office. The contents of the aforesaid Patent Application are incorporated herein for all purposes.

TECHNICAL FIELD

The present invention relates to a method, an apparatus and a computer readable storage medium with instructions for monitoring a movement of a transportation device. The invention further relates to a method, a device and a computer readable storage medium with instructions for planning a movement of a transportation device, as well as a transportation device using such a method or such a device.

BACKGROUND

For years, there can be seen an increasing spread of digital technologies in society. It is to be expected that this digitization of society will increase significantly in the everyday life. Virtual reality and augmented reality technologies and applications represent a special form of digitization. Augmented Reality (AR) is the enrichment of the real world with virtual elements, which regarding location are correctly registered in the three-dimensional space and allow real-time interaction. Synonymously, the term “mixed reality” is used. For the display of AR representations for example augmented reality glasses can be used. Augmented reality glasses are worn like normal glasses, but comprise one or more projection units or displays, with the help of which information can be projected in front of the eyes or directly onto the retina of the wearer of the glasses. The glasses are designed in such a manner that the wearer can also realize the environment. An increasing acceptance of this devices may be expected due to new technologies for augmented reality glasses, for example light field technology, advances in battery technology as well as the entry of large companies into this.

Alternatively, AR displays can also be generated by using augmented reality smartphones or augmented reality tablets. Thereby, virtual elements and the environment recorded by a camera of the respective device are together displayed on the display of the respective device. A projection of contents into the real environment is also possible.

Overall, it is foreseeable that augmented reality glasses, augmented reality smartphones, and other augmented reality devices will be increasingly being used by end customers. It can be expected that in the future this sector will offer powerful options, for example for providing user interfaces based on augmented reality representations in the automotive sector as the providers of the market-leading operating systems are currently making great efforts to establish their development environments and functions for augmented reality applications. One possible application of such user interfaces are movement maneuvers of a transportation device. A situation is considered in which the driver or another person is outside a vehicle that is currently performing a maneuver. For example, this relates to scenarios of use in which the vehicle carries out automatic parking maneuvers or, after the handover in corresponding zones, performs automated valet parking journeys. In the manual area the focus is on scenarios, which include manual parking or maneuvering.

SUMMARY

An object exists to provide an improved monitoring and planning of a movement of a transportation device.

The object is achieved by a method, by a device, and by a transportation device according to the independent claims. Embodiments of the invention are discussed in the dependent claims and the following description.

DESCRIPTION

In a first exemplary aspect, a method for monitoring a movement of a transportation device may comprise the steps:Receiving of information about a trajectory of the transportation device;Displaying the trajectory on a display unit of a mobile device in form of an augmented reality representation;Detecting an input of a user of the mobile device for influencing the trajectory; andTransmitting information to the transportation device on basis of the input of the user.

Correspondingly, a computer-readable storage medium may contain instructions which when carried out by a computer allow the computer to perform the following steps for monitoring the movement of a transportation device:Receiving information about a trajectory of the transportation device;Displaying the trajectory on a display unit of a mobile device in the form of an augmented reality representation;Detecting an input of a user of the mobile device for influencing the trajectory; andTransmitting information to the transportation device based on the input of the user.

The term computer should be understood broadly. For example, it also includes mobile processor-based data processing devices.

Analogously, a device for monitoring a movement of a transportation device may comprise:a receiving unit for receiving information about a trajectory of the transportation device;a graphics unit for generating a representation of the trajectory on a display unit of a mobile device in the form of an augmented reality representation;a user interface detecting an input of a user of the mobile device for influencing the trajectory; anda transmission unit for transmitting information to the transportation device based on the input of the user.

In a second exemplary aspect, a method for planning a movement of a transportation device may comprise the steps:Transmitting information about a trajectory of the transportation device to a mobile device;Receiving information for influencing the trajectory from the mobile device based on an input of a user of the mobile device; andPlanning the movement of the transportation device based on at least the received information.

Correspondingly, a computer-readable storage medium may contain instructions which when carried out by a computer allow the computer to perform the following steps for planning a movement of a transportation device:Transmitting information about a trajectory of the transportation device to a mobile device;Receiving information for influencing the trajectory from the mobile device based on an input of a user of the mobile device; andPlanning the movement of the transportation device based on at least the received information.

The term computer should be understood broadly. For example, it also includes control devices in vehicles.

Analogously to this, a device for planning a movement of a transportation device may comprise:a transmission unit for transmitting information about a trajectory of the transportation device to a mobile device;a receiving unit for receiving information for influencing the trajectory from the mobile device based on an input of a user of the mobile device; anda movement planner for planning the movement of the transportation device based on at least the information received.

The solution of the teachings herein is based on the idea that information about the current and planned trajectory based on a current position may be transmitted from a transportation device to a mobile device and can be displayed by means of augmentations, for example by using a camera function of the mobile device. For example, wireless transmission paths such as Wi-Fi or Bluetooth may be used for transmission. In this manner, an outside positioned person, for example the driver of a vehicle, may point the camera of his smartphone at the vehicle and is then able to see augmented displays of the planned driving path on the display of his smartphone. Alternatively, he may view the vehicle through his augmented reality glasses, which then generate the necessary augmentations. In contrast to known approaches, where the user can only start or stop a movement of the vehicle, the user now may actively influence the trajectory of the transportation device. In this manner, the user is able to react on potentially dangerous situations and has greater control over the movement of the transportation device. In the automotive sector the teachings described are helpful for both classic car applications as well as for usage scenarios with heavy, for example multi-axle, commercial vehicles with particularly high requirements for shunting operations. The solution beneficially may be used also for movement systems for caravans, which may be controlled for example by means of a remote control.

In some embodiments, the mobile device receives information about obstacles detected by a sensor system of the transportation device and shows them on the display unit. By the augmented reality representation, it may also be visualized whether the transportation device did correctly recognize certain static or dynamic obstacles in the environment, for example concrete pillars, the supports of a carport or people. For example, at initial operations fear may be taken from the user, especially from the inexperienced users, that the transportation device is self-damaged, for example due to sensor errors.

In some embodiments, the user's input determines a destination for the transportation device, a path for the transportation device, or information about an obstacle. For example, the use of a touch-sensitive displays qualifies for this. A target point or an obstacle may be determined for example by simply tapping the display at the appropriate position. A path in turn may be determined by drawing the path on the display with a finger. The inputs mentioned may thus be made in very intuitive and particularly easy manner. Additionally, dedicated control surfaces may be provided, for example for starting or stopping the movement. The function of a dead man's button may also be implemented via such a control surface.

In some embodiments, the user marks an obstacle on the display unit or classifies a marked obstacle. For example, the user may tap or encircle an obstacle on a touch-sensitive display. Unless the marked element is not automatically recognized after manual marking, an assignment of meaning may be made possible. For this purpose, the user may use for example a drop-down menu, in which different classifications are available. This may be location-based classifications such as “obstacle on the ground”, “vertical obstacle”, or “dynamic object”. However, more precise classifications are also possible, such as “pothole”, “person” or “wall”. The user may thus easily add or correct obstacles that are not or incorrectly recognized by the sensor system. This information may also be used to improve the data situation on the part of the perception of the environment of the transportation device.

In some embodiments, the transportation device receives environmental information or information from other sources captured by the mobile device and takes them into account in planning the movement. A possibility to effectively integrate the mobile device consists in using the camera image of the mobile device as an additional sensor and to provide the contents to the sensor fusion of the transportation device.

Here, e.g., two ways are conceivable. On the one hand, the camera image may be provided to the evaluation units of the transportation device without further information. In this case, the corresponding technical software module may perform an improvement or validation of the own perception of the environment by the on-board algorithms, for example, because distances may be better assigned or because objects may only be discovered through the new perspective.

On the other hand, image processing steps may be executed already on the part of the mobile device or server based. Here, essential processing steps may then be carried out for measurement and detection of the environment, which anyway have to be carried out to display augmented reality representations. The results may then passed on to the modules of the transportation device in aggregated form.

Of course, it is also possible to use the approaches described above in parallel. Since mobile devices are also equipped with microphones, according to the principles described above, acoustic signals may also be provided to the transportation device and may be used to interpret the current situation. As for example tire movements generate characteristic sound patterns on different surfaces, it is for example possible to respond with measures to unexpected noise, such as glass shattering or collision noise. For example, a parking process may be stopped, a trajectory may be changed, or a speed may be reduced, etc.

In some embodiments, a method according to some embodiments or a device according to some embodiments is used for planning a movement in a transportation device. The transportation device for example may be a motor vehicle. However, the solution according to the teachings herein is also useable for other types of transportation devices, such as for example for ships, aircraft, especially when moving on the ground, or automated transport systems.

In the shipping sector, the teachings described enable better forecasting of the movements and positions of ships, both in manual and in automatic operation. In this manner local pilots, but also the crew and the passengers of a ship, with the help of their mobile device may recognize and predict the effects of the current rudder angle (course) and machine power (speed) related to the actual fairway and the rest of the environment.

As for the maritime sector, the teachings presented are also highly relevant in the field of air travel. On the one hand, deployments by pilots are conceivable here, in which especially on the ground (taxiing from the gate to the runway or from the runway to the gate) optical orientation may be given for the correct path. On the other hand, a mobile device may also be used by the ground staff, for example by a park guide or the airfield surveillance, to display the current trajectory of an aircraft. For example, the marshaller would point a mobile device at an aircraft and thereby see the current trajectory and possibly also the continuing path to the finish gate.

The same principle may also be applied to rail and bus transport. In this case also, the users may be on the one hand the customers, but on the other hand also the rail or railroad staff standing on the bus platform. In this case also, the goal is to understand, to predict and, if desired, to control the movement and the purpose of another vehicle.

Another case of use is the use in an automated container terminal, where automated lifting vehicles are used and may be monitored. In such a manner, the security in dealing with such heavy-duty vehicles may be increased significantly.

Another use case is the use by driving schools. Here, driving inputs of a student may be translated into a future augmented reality trajectory. On the one hand this information may then be used by the driving instructor for giving feedback. However, for the student himself a recording may also be relevant later as a base for feedback.

Further features of the present invention will become apparent from the following description and the appended claims in connection with the FIGS.

For a better understanding of the principles of the present invention, the following embodiments are explained in more detail with reference to the FIGS. Of course, the invention is not limited to these embodiments and the features described may also be combined or modified without leaving the scope of the invention as defined in the appended claims.

Specific references to components, process steps, and other elements are not intended to be limiting. Further, it is understood that like parts bear the same or similar reference numerals when referring to alternate FIGS. It is further noted that the FIGS. are schematic and provided for guidance to the skilled reader and are not necessarily drawn to scale. Rather, the various drawing scales, aspect ratios, and numbers of components shown in the FIGS. may be purposely distorted to make certain features or relationships easier to understand.

FIG.1schematically shows a method for monitoring a movement of a transportation device. First, information about a trajectory of the transportation device are received from a mobile device10.

Further, information about obstacles may be received, that have been detected by a sensor system of the transportation device. The trajectory and possibly the detected obstacles are then displayed11on a display unit of the mobile device in the form of an augmented reality representation. Thereupon, an input of a user of the mobile device for influencing the trajectory is recorded12. For example, the input of the user may determine a point of destination for the transportation device, a path for the transportation device or an information about an obstacle. To do so, for example, the user marks an obstacle on the display unit or classifies a marked obstacle. Finally, based on the user's input information is transmitted13to the transportation device. In doing so, additional information about the environment detected by the mobile device may be transmitted to the transportation device.

FIG.2schematically shows a first embodiment of a device20for monitoring a movement of a transportation device. The device20has an input21, by which a receiving unit22receives, for example, information about a trajectory of the transportation device. In addition, information about obstacles detected by a sensor system of the transportation device may be received. The device20further comprises a graphics unit23for generating a representation of the trajectory and of the obstacles, that have been recognized, on a display unit of a mobile device90in the form of an augmented reality representation. An input of a user for influencing the trajectory may be detected by a user interface28. The input of the user for example may be a point of destination for the transportation device, a path for the transportation device or information about an obstacle. To do this, the user for example marks an obstacle on the display unit or classifies a marked obstacle. Based on the input of the user a transmission unit26transmits via an output27of the device20information to the transportation device. In addition, information about the environment recorded by the mobile device may be transmitted to the transportation device.

The receiving unit22, the graphics unit23and the transmission unit26may be controlled by a control unit24. Via the user interface28, settings of the receiving unit22may possibly be changed by the graphics unit23, the transmission unit26or the control unit24. Date accumulating in the device may be stored in a memory25of the device20, if necessary, for example for later evaluation or for use by the components of the device20. The receiving unit22, the graphics unit23, the transmission unit26and control unit24may be implemented as dedicated hardware, for example as integrated circuits. However, of course they may partially or fully be combined or implemented as software running on a suitable processor, for example on a GPU or a CPU. The input21and the output27may be implemented as separate interfaces or as a combined bidirectional interface. The device20may be a stand-alone component that is connected to the mobile device via a data connection. However, it may also be integrated into the mobile device.

FIG.3schematically shows a second embodiment of a device30for monitoring a movement of a transportation device. The device30comprises a processor32and a memory31. For example, the device is a mobile computing unit, for example in a tablet or in a smartphone. Instructions are stored in the memory31that, when executed by the processor32, cause the device30to carry out the steps according to one of the methods described. The instructions stored in the memory31thus incorporate a program executable by the processor32which implements the method according to the present embodiment. The device30comprises an input33for receiving information. Data generated by the processor32is provided via an output34. Further, they may be stored in the memory31. The input33and the output34may be combined in a bidirectional interface.

The processor32may comprise one or more processor units, for example microprocessors, digital signal processors, or combinations thereof.

The memories25,31of the described embodiments may comprise volatile as well as non-volatile storage areas and may comprise a wide variety of storage devices and storage media, for example, hard drives, optical storage media or semiconductor memory.

FIG.4schematically shows a method for planning a movement of a transportation device. First, information about a trajectory of the transportation device is transmitted to a mobile device15. Additionally, information about obstacles recognized by a sensor system of the transportation device may be transmitted. Then, information transmitted by the mobile device is received16for influencing the trajectory based on an input of a user of the mobile device. The input of the user for example may determine a point of destination for the transportation device, a path for the transportation device or information about an obstacle. Finally, the movement of the transportation device is planned17based on at least the received information. In addition, the transportation device may receive16environmental information detected by the mobile device or information from other sources and take them into account17when planning the movement.

FIG.5schematically shows a first embodiment of a device40for planning a movement of a transportation device. The device40has an output47via which a communication unit46transmits information about a trajectory of the transportation device to a mobile device. Additionally, information about obstacles detected by a sensor system of the transportation device may be transmitted thereby. Via an input41of device40, a receiving unit42receives information transmitted by the mobile device for influencing the trajectory based on an input of a user of the mobile device. The input of the user for example may determine a point of destination for the transportation device, a path for the transportation device or information about an obstacle. Based on at least the information received a movement planner43plans the movement of the transportation device. Additionally, when planning the movement, the motion planner43may consider information captured by the mobile device or information from other sources detected by the mobile device, which were received by the receiving unit42. The planning data of the movement planner43may be output via output47to control units of the transportation device.

The receiving unit42, the movement planner43and the transmission unit46may be controlled by a control unit44. As the case may be, settings of the receiving unit42, the movement planner43, the transmission unit46or the control unit44may be changed via a user interface48. Data accumulating in the device40may be stored in a memory45of the device40, if necessary, for example for a later evaluation or for a use by the components of the device40. The receiving unit42, the movement planner43, the transmission unit46and the control unit44may be realized as dedicated hardware, for example as integrated circuits. However, of course they may partially or fully be combined or implemented as software running on a suitable processor, for example on a GPU or a CPU. The input41and the output47may be implemented as separate interfaces or as a combined bidirectional interface.

FIG.6schematically shows a second embodiment of a device50for planning a movement of a transportation device. The device50comprises a processor52and a memory51. For example, the device50is a control unit of a transportation device. Instructions are stored in the memory51that, when executed by the processor52, cause the device50to carry out the steps according to one of the procedures described. The instructions stored in the memory51thus incorporate a program executable by the processor52which implements the method according to the present embodiment. The device50comprises an input53for receiving information. Data generated by the processor52is provided via an output54. Further, they may be stored in the memory51. The input53and the output54may be combined in a bidirectional interface.

The processor52may comprise one or more processor units, for example microprocessors, digital signal processors, or combinations thereof.

The memories45,51of the described embodiments may comprise volatile as well as non-volatile storage areas and may comprise a wide variety of storage devices and storage media, for example, hard drives, optical storage media or semiconductor memory.

FIG.7schematically shows a smartphone60as an example of a mobile device suitable to be used with a method or a device according to the present embodiment. The smartphone60has a display62that is arranged to display an augmented reality representation. For this purpose, by a camera63the smartphone60records an environment which is displayed on the display62. Information about superimposing the displayed environment on the display62is received via an interface61. In particular, data may be exchanged with a transportation device via the interface61. The display62is designed to be touch-sensitive and serves as an interface for interaction with a user.

FIG.8schematically shows augmented reality glasses70as an example of a mobile device, which is suitable for use with a method or a device according to the present embodiment. The augmented reality glasses70comprise a camera71for detecting environmental data of an environment of the augmented reality glasses. A pose of the augmented reality glasses may be determined by means of a pose determination device72, for example based on the environmental data. In accordance with the determined pose a graphics unit73generates a representation for the augmented reality glasses. For this purpose, the pose determination device72and the graphics unit73may exchange data with each other. Information about content to be displayed is received via an interface74. In particular, data exchange with a transportation device may take place via the interface74. The graphics unit73The representation generated by the graphics unit73is displayed by a projection unit75. In the example inFIG.8a monocular display is used, i.e. in front of only one eye of the user a display is arranged. Of course, the solution described herein may also be used with augmented reality glasses with a binocular display with one display per eye or with a binocular display where both eyes have a common display. The interaction with the user may performed by using voice commands or by using an additional input device.

FIG.9schematically shows a transportation device80in which a solution according to the present embodiment is realized. In this example, the transportation device80is a motor vehicle. The motor vehicle comprises an imaging sensor81, for example a camera. Further, a radar sensor82and a lidar sensor83are present. Further components of the motor vehicle are a navigation device84and a data transmission unit85. By means of the data transmission unit85, a connection to a mobile device or to a service provider may be established, especially for exchange of information for the planning of movement. The data of the various sensors81,82,83as well as information received by the data transmission unit85for the planning of movement are considered by a device40according to the present embodiment when planning the movement of the motor vehicle. A memory86is provided for storing data. The data exchange between the various components of the motor vehicle takes place via a network87.

FIG.10schematically shows a system diagram of the solution according to the present embodiment. There are shown the transportation device80, here in the form of a motor vehicle, and the mobile device90, which exchange data over a communication link. Additionally, there are shown a further sensor96in the form of an observation camera and a backend97. According to the previous description the monitoring and the planning of the movement were performed by using the sensor technology present in the mobile device90or in the transportation device80, the sensor data recorded there and the computing capacity available there.

An extension is possible if a backend97is available as a wireless data connection. With the help of such a backend, sensors of other nearby vehicles, mobile devices, or permanently installed sources96may be made available to the mobile device90or the transportation device80as a data source. The augmentation or the trajectory is then computed based on the own and foreign sensor data. To prevent harmful attack scenarios, the communication for example is encrypted and provided with a unique time stamp for each data packet.

If the environmental data of the own mobile device90and, as the case may be, of the means of transport80are not sufficient, e.g. due to the camera image being covered by a pillar, environmental data stored in the backend97may be used for tracking or object recognition. For this, for example a time stamp for each date stored in the backend97has to be provided so that it may be decided whether this data may or may not serve as a source for each application.

Data that were delivered at other times by mobile devices or transportation device to the backend97, may be provided as an additional alternative to the user by means of an augmentation so that an added value is offered for the user. An example of this is information like “In the parking space you chose today there were already parking X people” or “The vehicle parking three spaces to the left of you usually leaves in 5 minutes”. Thereby, of course, data protection has to be ensured.

FIG.11illustrates a visualization of a trajectory92and of objects93recognized by sensors in the form of an augmented reality representation on a display unit91of a mobile device90. In this manner, an outside standing person, for example a driver having left, may understand the actions of an automatically driving vehicle and intervene for correction. In this manner, the actions of a manually driving person may also be traced from the outside, for example use a probably route of the the current steering wheel angle and for example, for a more precise instruction.

FIGS.12to15illustrate the active influence of a user on the planning of a movement, here using the example of a motor vehicle. The user directs the camera of his mobile device onto the vehicle and works with the augmented representations or generates them. Only displayed content is shown on the display unit of the mobile device.

FIG.12shows the planned trajectory92of a vehicle in the direction of a parking area. In the parking area there is an obstacle93in the form of a pothole, which has not yet been recognized by the sensor system of the vehicle.

InFIG.13, the user marks the pothole with his finger on the camera image of the mobile device. The corresponding information is then transmitted from the mobile device to the vehicle, which may now consider the identified pothole in the planning of the movement. Unless the selected element after having been manually selected is automatically recognized, the user for his part may perform an assignment of meaning.

FIG.14shows that the pothole has been recognized by the vehicle as an obstacle93. In the specific example, however, the movement planned by the movement planner was not changed as the pothole had not been considered a problematic obstacle93. The user would therefore like to assign a route to the vehicle himself. For this task, which is shown inFIG.15, in a first step the user may draw a simple path95with the finger on the camera image of the mobile device. In a second step, this path may then be checked by the corresponding path planning unit of the transportation device and be implemented in an executable proposal. For this purpose, for example a new and improved augmented representation reflecting the actual capabilities of the vehicle is provided. After an optional third step, the confirmation, this new path then is driven.

LIST OF REFERENCE NUMERALS

10Receiving information about a trajectory of a transportation device11Show the trajectory as an augmented reality representation12Detecting input from a user13Transmission of information about the user's input to the transportation device15Transmission of information about a trajectory of the transportation device16Receiving of information for influencing the trajectory17Planning of the movement of the transportation device20Device21Input22Receiving unit23Graphics unit24Control unit25Storage26Transmission unit27Output28User interface30Device31Storage32Processor33Input34Output40Device41Input42Receiving unit43Movement planner44Control unit45Storage46Transmission unit47Output48User interface50Device51Storage52Processor53Input54Output60Smartphone61Interface62Display63Camera70Augmented reality glasses71Camera72Interface73Pose determination device74Graphics unit75Projection unit80Transportation device81Imaging sensor82Radar sensor83Lidar sensor84Navigation device85Data transmission unit86Storage87Network90Mobile device91Display unit92Trajectory93Obstacle94Target point95Path96Sensor97Backend