Systems and methods for estimating velocity of an autonomous vehicle and state information of a surrounding vehicle

Systems and methods for estimating velocity of an autonomous vehicle and state information of a surrounding vehicle are provided. In some aspects, the system includes a memory that stores instructions for executing processes for estimating velocity of an autonomous vehicle and state information of the surrounding vehicle and a processor configured to execute the instructions. In various aspects, the processes include: receiving image data from an image capturing device; performing a ground plane estimation by predicting a depth of points on a road surface based on an estimated pixel-level depth; determining a three-dimensional (3D) bounding box of the surrounding vehicle; determining the state information of the surrounding vehicle based on the ground plane estimation and the 3D bounding box; and determining the velocity of the autonomous vehicle based on an immovable object relative to the autonomous vehicle. In some aspects, an operation of the autonomous vehicle may be controlled based on at least one of the state information or the velocity of the autonomous vehicles.

TECHNICAL FIELD

This disclosure relates to methods and systems for estimating velocity of an autonomous vehicle and state information of a surrounding vehicle.

BACKGROUND

Understanding ego-motion, e.g., the velocity of an autonomous vehicle, and state information of a surrounding vehicle is essential for operating autonomous vehicles and advanced driver-assistance systems (ADAS) enabled vehicles. For example, 3D position, velocity, and orientation of surrounding vehicles are critical information for decision making and path planning for operating autonomous vehicles and advanced driver-assistance systems (ADAS) enabled vehicles. Furthermore, for autonomous vehicles to be widely accepted, these systems may be as simple as possible to ease implementation and ensure reliability while minimizing cost.

Many systems for estimating ego-motion and surrounding vehicle state may rely on, for example, LiDAR or multiple sensors, such as a combination of two or more of a LiDAR, a camera, and a radar. Other systems may rely on cameras to make a vehicle sensor system cost-effective and straightforward. However, it may be still challenging to estimate ego-motion and state information of a surrounding vehicle with information from only a monocular camera compared to information from multiple sensors. 2D object detection Al-based algorithms have achieved great performance with fast and accurate 2D object detection using a monocular camera. However, the 2D object detection results lack distance information. On the other hand, LiDAR and stereo camera are generally used in autonomous vehicle development to estimate 3D features of the vehicles. However LiDAR technology may be expensive, may not provide the long-term reliability required in automotive applications due to the existence of rotating parts. Stereo camera may also expensive, may require high precision calibration. The monocular 3D object detection may be based on regression of a 3D bounding box in a 2D image, or may be based on a fixed single ground plane, which is not constant in driving situations.

SUMMARY

In one aspect, the present disclosure is related to a system for estimating velocity of an autonomous vehicle and state information of a surrounding vehicle. The system may include a memory that stores instructions for executing processes for estimating the velocity of the autonomous vehicle and state information of the surrounding vehicle and a processor configured to execute the instructions. The processes may include: receiving image data from an image capturing device; performing a ground plane estimation by predicting a depth of points on a road surface based on an estimated pixel-level depth; determining a three-dimensional (3D) bounding box of the surrounding vehicle; determining the state information of the surrounding vehicle based on the ground plane estimation and the 3D bounding box; and determining the velocity of the autonomous vehicle based on an immovable object relative to the autonomous vehicle. In some aspects, an operation of the autonomous vehicle may be controlled based on at least one of the state information or the velocity of the autonomous vehicles.

In another aspect, the present disclosure is related to a method for estimating velocity of an autonomous vehicle and state information of a surrounding vehicle. The method may include: receiving image data from an image capturing device; performing a ground plane estimation by predicting a depth of points on a road surface based on an estimated pixel-level depth; determining a three-dimensional (3D) bounding box of the surrounding vehicle; determining the state information of the surrounding vehicle based on the ground plane estimation and the 3D bounding box; and determining the velocity of the autonomous vehicle based on an immovable object relative to the autonomous vehicle. In some aspects, an operation of the autonomous vehicle may be controlled based on at least one of the state information or the velocity of the autonomous vehicles.

In a further aspects, the present disclosure is related to a non-transitory computer-readable storage medium containing executable computer program code for estimating velocity of an autonomous vehicle and state information of a surrounding vehicle. The code may comprise instructions configured to cause a processor to: receive image data from an image capturing device; perform a ground plane estimation by predicting a depth of points on a road surface based on an estimated pixel-level depth; determine a three-dimensional (3D) bounding box of the surrounding vehicle; determine the state information of the surrounding vehicle based on the ground plane estimation and the 3D bounding box; and determine the velocity of the autonomous vehicle based on an immovable object relative to the autonomous vehicle. In some aspects, an operation of the autonomous vehicle may be controlled based on at least one of the state information or the velocity of the autonomous vehicles.

DETAILED DESCRIPTION

The following includes definitions of selected terms employed herein. The definitions include various examples and/or forms of components that fall within the scope of a term and that may be used for implementation. The examples are not intended to be limiting.

A “processor,” as used herein, processes signals and performs general computing and arithmetic functions. Signals processed by the processor may include digital signals, data signals, computer instructions, processor instructions, messages, a bit, a bit stream, or other computing that may be received, transmitted and/or detected.

A “bus,” as used herein, refers to an interconnected architecture that is operably connected to transfer data between computer components within a singular or multiple systems. The bus may be a memory bus, a memory controller, a peripheral bus, an external bus, a crossbar switch, and/or a local bus, among others. The bus may also be a vehicle bus that interconnects components inside a vehicle using protocols, such as Controller Area network (CAN), Local Interconnect Network (LIN), Automotive Ethernet, among others.

A “memory,” as used herein may include volatile memory and/or non-volatile memory. Non-volatile memory may include, for example, ROM (read only memory), PROM (programmable read only memory), EPROM (erasable PROM) and EEPROM (electrically erasable PROM). Volatile memory may include, for example, RAM (random access memory), synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), Graphic Dynamic RAM (GDRAM), and/or direct RAM bus RAM (DRRAM).

An “operable connection,” as used herein may include a connection by which entities are “operably connected”, is one in which signals, physical communications, and/or logical communications may be sent and/or received. An operable connection may include a physical interface, a data interface and/or an electrical interface.

A “vehicle,” as used herein, refers to any moving vehicle that is powered by any form of energy. A vehicle may carry human occupants or cargo. The term “vehicle” includes, but is not limited to: cars, trucks, vans, minivans, SUVs, motorcycles, scooters, boats, personal watercraft, and aircraft. In some cases, a motor vehicle includes one or more engines.

Generally described, the present disclosure provides systems and methods for estimating velocity of an autonomous vehicle and state information of a surrounding vehicle. For example, in some implementations, the present disclosure provides systems and methods for estimating velocity of an autonomous vehicle and state information of a surrounding vehicle by combining 3D bounding box detection, depth estimation, ground plane estimation, and flow estimation.

Turning toFIG. 1, a schematic view of an example operating environment100of a vehicle system110according to an aspect of the disclosure is provided. The vehicle system110may reside within a vehicle102. The components of the vehicle system110, as well as the components of other systems, hardware architectures, and software architectures discussed herein, may be combined, omitted or organized into various implementations.

The vehicle102may generally include an electronic control unit (ECU)112that operably controls a plurality of vehicle systems. The vehicle systems may include, but are not limited to, the vehicle system110, among others, including vehicle HVAC systems, vehicle audio systems, vehicle video systems, vehicle infotainment systems, vehicle telephone systems, and the like. The vehicle system110may include a front camera or other image-capturing device (e.g., a scanner)120, roof camera or other image-capturing device (e.g., a scanner)121, and rear camera or other image capturing device (e.g., a scanner)122that may also be connected to the ECU112to provide images of the environment surrounding the vehicle102. The vehicle system110may also include a processor114and a memory116that communicate with the front camera120, roof camera121, rear camera122, communications device130, and driving system132.

The ECU112may include internal processing memory, an interface circuit, and bus lines for transferring data, sending commands, and communicating with the vehicle systems. The ECU112may include an internal processor and memory, not shown. The vehicle102may also include a bus for sending data internally among the various components of the vehicle system110.

The vehicle102may further include a communications device130(e.g., wireless modem) for providing wired or wireless computer communications utilizing various protocols to send/receive electronic signals internally with respect to features and systems within the vehicle102and with respect to external devices. These protocols may include a wireless system utilizing radio-frequency (RF) communications (e.g., IEEE 802.11 (Wi-Fi), IEEE 802.15.1 (Bluetooth®)), a near field communication system (NFC) (e.g., ISO 13157), a local area network (LAN), a wireless wide area network (WWAN) (e.g., cellular) and/or a point-to-point system. Additionally, the communications device130of the vehicle102may be operably connected for internal computer communication via a bus (e.g., a CAN or a LIN protocol bus or/and Automotive Ethernet) to facilitate data input and output between the electronic control unit112and vehicle features and systems. In an aspect, the communications device130may be configured for vehicle-to-everything (V2X) communications. For example, V2X communications may include wireless communications over a reserved frequency spectrum. As another example, V2X communications may include an ad hoc network between vehicles set up using Wi-Fi or Bluetooth®.

The vehicle102may include a camera120, such as a monocular camera. For example, the camera120may be a digital camera capable of capturing one or more images or image streams that may be provided to a driving system132or a remote server, such as a manufacturer system, as discussed with respect toFIG. 3. The driving system132may also include a memory that stores instructions for executing processes for estimating velocity of an autonomous vehicle and state information of a surrounding vehicle, and a processor configured to execute the instructions.

According to aspects of the present disclosure, as illustrated inFIG. 2, the driving system132may be configured to receive an input205, such as images or a data stream from the camera120, and to execute a plurality of neural networks, namely a depth network210A, a 3D bounding box network210B, and a flow network210C. In some implementations, the depth network210A, the 3D bounding box network210B, and the flow network210C may be used to generate a plurality of outputs, namely state information of a surrounding vehicle230and ego-motion235. The state information of a surrounding vehicle230may include a 3D position230A of a surrounding vehicle, an orientation230B of the surrounding vehicle, and a velocity230C of the surrounding vehicle.

In some implementations, the depth network210A may be configured to generate pixel-level depth of a road surface traversed by the vehicle102. For example, in some implementations, the depth network210A may be a deep-learning model pre-trained on a KITTI dataset, as should be understood by those of ordinary skill in the arts. As such, the depth network210A may be configured to perform a ground plane estimation predicting a depth of points on the road surface based on the estimated pixel-level depth. In some implementations, the depth network210A may implement unsupervised learning, and therefore, the depth network210A may not take into consideration factors such as variations of ground truth annotations, thereby eliminating costs associated with the annotating images. The depth network210A may be configured to provide the depth information of the points on the road surface to a ground plane estimator215.

According to aspects of the present disclosure, the 3D bounding box network210B may be configured to determine a 3D bounding box of another vehicle surrounding the vehicle102(interchangeably referred to herein as a “surrounding vehicle”) using a machine learning algorithm. In some implementations, the machine learning algorithm may be generated using a fully convolutional network (FCN) framework and a multi-scale network based on a Single Shot MultiBox Detector (SSD) and a multi-scale convolutional neural network (MS-CNN) and Dense Convolutional Network (DenseNet). Using these machine learning algorithms, the 3D bounding box network210B may implement a 3D bounding box representation, which is independent of an image projection matrix. Using the 3D bounding box, the 3D bounding box network210B may be configured to generate the 3D Position230A of surrounding vehicles in a 2D image. For example, the 3D Position230A may be coordinates of three bottom vertices and a height of a bounding box of the surrounding vehicle. According to some aspects of the present disclosure, each bottom vertex coordinate [u v]Tmay be reconstructed in the 3D world using a projection matrices (1) and (2) and a ground plane equation (3).

P=[fx0cx0fycy001]⁡[r11r1⁢2r1⁢3t1r21r2⁢2r2⁢3t2r31r3⁢2r3⁢3t3](1)[xyz1]=P-1⁡[uν1](2)ax+b⁢y+c⁢z+d=0(3)
where f is a focal length, c is a camera center, r is a rotation factor, and t is a translation factor. That is, the 3D bounding box network210B may be configured to generate 3D coordinates for the top vertices of the bounding box of detected vehicles using a height output and estimate the 3D bounding box.

According to further aspects of the present disclosure, the ground plane estimator215may receive the depth information of the points on the road surface and the 3D bounding box from the depth network210A and the 3D bounding box network210B, respectively, and may be configured to perform a ground plane correction using this information. For example, while the vehicle102is moving, ground plane coefficients may be continuously changing depending on the inclination of the road surface, as well as pitching, rolling, and yaw angle of the vehicle102. To account for the changes in the ground plane coefficients, the ground plane estimator215may be configured to update the ground plane coefficients by measuring an actual inclination of the ground surface in real-time. To achieve this, the ground plane estimator215may be configured to determine ground plane coefficients using a RANSAC (Random Sample Consensus) algorithm for fitting to optimal coefficients (a, b, c, d) in equation (3) using the bottom four corners vertices of the 3D bounding box. As such, the ground plane coefficients may be refined to fit to the road surface in real-time, and in order to solve this problem, the ground plane estimator215may be configured to use the depth information from the depth network210A. In some implementations, the depth information from the depth network210A may be a normalized depth and the ground plane estimator215may be configured to convert the normalized depth to an actual distance. For example, the ground plane estimator215may convert the normalized depth into an actual distance based on the known distance beforehand, as illustrated in equation (4).
Dist=k×Depth  (4)
where k is the coefficient between the normalized depth (Depth) and actual distance (Dist).

In further implementations, the ground plane estimator215may be configured to select a plurality of fixed points in a lower portion of the image of the input205that includes the road surface. After that, the ground plane estimator215may be configured to remove one or more points of the plurality of fixed points that are inappropriate for estimating the ground plane. To determine which points are inappropriate, the ground plane estimator215may determine which points are outside of the 3D bounding box for every frame. Using the information from the depth network210A, the ground plane estimator215may be configured to project the remaining points into a 3D world coordinate system and to execute the RANSAC algorithm to estimate the corrected ground plane coefficients. To update the ground plane, the ground plane estimator215may be configured to determine whether the update is possible by comparing the initial ground plane coefficients, ground plane coefficients in a previous frame, and currently estimated ground plane coefficients based on equation (5). When the update is determined to be possible, the ground plane estimator215may be configured to compute new coefficients of the ground plane based on equation (6).

In some implementations, the 3D bounding box network210B may be further configured to determine the orientation230B of the surrounding vehicle in the 3D world coordinate system. For example, the orientation230B may be calculated by projecting the predicted 3D bounding box from the 3D bounding box network210B into the corrected ground plane from the ground plane estimator215using the depth estimation from the depth network210A.

In accordance with aspects of the present disclosure, the flow network210C may be configured to assess the surroundings of the vehicle102. For example, the flow network210C may be configured to determine an absolute velocity of the vehicle102and the relative velocity of the surrounding vehicle, and using this information, the flow network210C may then calculate the absolute velocity of the surrounding vehicle. In some aspects, the flow network210C may be a model pre-trained using the KITTI dataset, as should be understood by those ordinary skill in the arts.

The flow network210C may include an ego flow estimator225for determining the velocity, such as an absolute velocity, of the vehicle102. For example, the ego flow estimator225may be configured to determine the absolute velocity of the vehicle102based on the flow of immovable objects, such as the ground. For example, in order to estimate the velocity of the vehicle102, the ego flow estimator225may be configured to assume that a road surface as close as possible to the vehicle102is a fixed calculation area405, as illustrated inFIG. 4. The ego flow estimator225may be configured to determine a 2D flow vector (u, v) in this fixed calculation area405and to extract the 2D flow into a 3D flow (flowGx, flowGy, flowGz) using the projection matrix (1), (2) and the ground plane (3) received from the 3D bounding box network210B. Using this information, the ego flow estimator225may be configured to compute a ground speed VGat the fixed calculation area405using (7).

The flow network210C may also include a flow extraction estimator220for determining the velocity of the surrounding vehicle. For example, using the 3D position of the surrounding vehicle, the flow extraction estimator220may be configured to calculate a relative velocity of the surrounding vehicle using the differential of position of the vehicle102and the surrounding vehicle. The flow extraction estimator220may calculate the absolute velocity of surrounding vehicle using the relative velocity of the surrounding vehicle and the absolute velocity of the ego-vehicle calculated by the ego flow estimator225. To achieve this, the flow extraction estimator220may be configured to project the 2D flow of the surrounding vehicle to the 3D coordinate of the surrounding vehicle, rather than the ground plane of the vehicle. As discussed above, the 3D bounding box network210B may be configured to estimate the 3D bounding box of the surrounding vehicle with the 3D coordinates of each vertex. In some implementations, the flow extraction estimator220may convert an arbitrary 2D point on the 3D bounding box505into a 3D position and generate a new plane A passing through the arbitrary point and parallel to the ground plane GP, as illustrated inFIG. 5. Additionally, the flow extraction estimator220may be configured to convert the 2D flow into 3D flow based on the plane A corresponding to the surrounding vehicle. For example, the 2D flow at an arbitrary point may be projected on the plane A rather than the ground plane GP.

In some aspects of the present disclosure, the flow extraction estimator220may be configured to account for different shapes of vehicles. To achieve this, the flow extraction estimator220may be configured to calculate the 2D flow in a lower half of the 3D bounding box505. Moreover, the flow extraction estimator220may use two vertical planes of the 3D bounding box505near the vehicle102for 2D flow extraction as the two vertical planes may be visible from the vehicle102when there are no occlusions caused by other obstacles. In still further aspects, a 2D flow vector (u, v) in the 3D bounding box505may be projected on each plane based on the 2D coordinates, and then the flow extraction estimator220may compute the 3D flow (flowSx, flowSy, flowSz) and the relative velocity of the surrounding vehicle using equation (8).

Vr_s⁡(Vr_Sx,Vr_Sy,Vr_Sz)=1m⁢Σi=1m⁢f⁢l⁢o⁢wsi⁡(x,y,z)(8)
where m is total pixel number in the lower half of the 3D bounding box. The flow extraction estimator220may be configured to convert the relative velocity to an absolute velocity. For example, in some implementations, flow extraction estimator220may calculate an absolute longitudinal velocity Va_szof the surrounding vehicle by subtracting the relative velocity of the surrounding vehicle Vr_szfrom the velocity VGZof the vehicle102based on equation 9:
Va_Sz=Vr_Sz+V′Gz(9)
Additionally, for the absolute lateral velocity, the flow extraction estimator220may be configured to recalculate a second lateral velocity VGx′ of the vehicle102near the surrounding vehicle from the based on calculated the lateral and longitudinal velocities VGx,VGzof the vehicle102. As illustrated inFIG. 6, a distance d0between RCego(e.g., a location on the vehicle102) and the camera center, e.g., a center of camera120) is constant for a given type of vehicle. As further illustrated inFIG. 6, a distance dGbetween the camera center and the center CGof the fixed ground plane, and the distance dSbetween the camera center and the center CSof the ground near the surrounding vehicle can be predicted using, for example, the ground plane estimator215and/or the depth network210A, as described herein. Using this information, the second lateral velocity VGx′ near the surrounding vehicle may be calculated based on equation (10):

Aspects of the present invention may be implemented using hardware, software, or a combination thereof and may be implemented in one or more computer systems or other processing systems. In an aspect of the present invention, features are directed toward one or more computer systems capable of carrying out the functionality described herein. An example of such a computer system800is shown inFIG. 8.

Computer system800includes one or more processors, such as processor804. The processor804is connected to a communication infrastructure806(e.g., a communications bus, cross-over bar, or network). Various software aspects are described in terms of this example computer system. After reading this description, it will become apparent to a person skilled in the relevant art(s) how to implement aspects of the invention using other computer systems and/or architectures.

Computer system800may include a display interface802that forwards graphics, text, and other data from the communication infrastructure806(or from a frame buffer not shown) for display on a display unit830. Computer system800also includes a main memory808, preferably random access memory (RAM), and may also include a secondary memory810. The secondary memory810may include, for example, a hard disk drive812, and/or a removable storage drive814, representing a floppy disk drive, a magnetic tape drive, an optical disk drive, a universal serial bus (USB) flash drive, etc. The removable storage drive814reads from and/or writes to a removable storage unit818in a well-known manner. Removable storage unit818represents a floppy disk, magnetic tape, optical disk, USB flash drive etc., which is read by and written to removable storage drive814. As will be appreciated, the removable storage unit818includes a computer usable storage medium having stored therein computer software and/or data.

Alternative aspects of the present invention may include secondary memory810and may include other similar devices for allowing computer programs or other instructions to be loaded into computer system800. Such devices may include, for example, a removable storage unit822and an interface820. Examples of such may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an erasable programmable read only memory (EPROM), or programmable read only memory (PROM)) and associated socket, and other removable storage units822and interfaces820, which allow software and data to be transferred from the removable storage unit822to computer system800.

Computer system800may also include a communications interface824. Communications interface824allows software and data to be transferred between computer system800and external devices. Examples of communications interface824may include a modem, a network interface (such as an Ethernet card), a communications port, a Personal Computer Memory Card International Association (PCMCIA) slot and card, etc. Software and data transferred via communications interface824are in the form of signals828, which may be electronic, electromagnetic, optical or other signals capable of being received by communications interface824. These signals828are provided to communications interface824via a communications path (e.g., channel)826. This path826carries signals828and may be implemented using wire or cable, fiber optics, a telephone line, a cellular link, a radio frequency (RF) link and/or other communications channels. In this document, the terms “computer program medium” and “computer usable medium” are used to refer generally to media such as a removable storage drive818, a hard disk installed in hard disk drive812, and signals828. These computer program products provide software to the computer system800. Aspects of the present invention are directed to such computer program products.

Computer programs (also referred to as computer control logic) are stored in main memory808and/or secondary memory810. Computer programs may also be received via communications interface824. Such computer programs, when executed, enable the computer system800to perform the features in accordance with aspects of the present invention, as discussed herein. In particular, the computer programs, when executed, enable the processor804to perform the features in accordance with aspects of the present invention. Accordingly, such computer programs represent controllers of the computer system800.

In an aspect of the present invention where the invention is implemented using software, the software may be stored in a computer program product and loaded into computer system800using removable storage drive814, hard drive812, or communications interface820. The control logic (software), when executed by the processor804, causes the processor804to perform the functions described herein. In another aspect of the present invention, the system is implemented primarily in hardware using, for example, hardware components, such as application specific integrated circuits (ASICs). Implementation of the hardware state machine so as to perform the functions described herein will be apparent to persons skilled in the relevant art(s).

FIG. 9illustrates a flowchart method for estimating a velocity of an autonomous vehicle and state information of a surrounding vehicle. The method900includes receiving image data from an image capturing device910. For example, the image data may be received from the camera120of the vehicle102. The method900also includes performing a ground plane estimation by predicting a depth of points on a road surface based on an estimated pixel-level depth920. For example, in some implementations, the depth network210A may be configured to generate pixel-level depth of a road surface traversed by the vehicle102. The depth network210A may be further configured to perform a ground plane estimation predicting a depth of points on the road surface based on the estimated pixel-level depth. In some implementations, the depth network210A may implement unsupervised learning, and therefore, the depth network210A may not take into consideration factors such as variations of ground truth annotations, thereby eliminating costs associated with the annotating images. The depth network210A may be configured to provide the depth information of the points on the road surface to a ground plane estimator215.

The method900also includes determining a three-dimensional (3D) bounding box of the surrounding vehicle930. For example, the 3D bounding box network210B may be configured to determine a 3D bounding box of another vehicle surrounding the vehicle102using a machine learning algorithm. In some implementations, the machine learning algorithm may be generated using a fully convolutional network (FCN) framework and a multi-scale network based on a Single Shot MultiBox Detector (SSD) and a multi-scale convolutional neural network (MS-CNN) and Dense Convolutional Network (DenseNet). Using these machine learning algorithms, the 3D bounding box network210B may implement a 3D bounding box representation, which is independent of an image projection matrix.

The method900further includes determining the state information of the surrounding vehicle based on the ground plane estimation and the 3D bounding box940. The state information of a surrounding vehicle230may include a 3D position230A of a surrounding vehicle, an orientation230B of the surrounding vehicle, and a velocity230C of the surrounding vehicle. Using the 3D bounding box, the 3D bounding box network210B may be configured to generate the 3D Position230A of surrounding vehicles in the 3D world coordinate system using the corrected ground plane from the ground plane estimator215using the depth estimation from the depth network210A. For example, the 3D Position230A may be coordinates of three bottom vertices and a height of a bounding box of the surrounding vehicle. According to some aspects of the present disclosure, each bottom vertex coordinate [u v]T may be reconstructed in the 3D world using a projection matrices (1) and (2) and a ground plane equation (3) shown above.

According to further aspects of the present disclosure, the ground plane estimator215may receive the depth information of the points on the road surface and the 3D bounding box from the depth network210A and the 3D bounding box network210B, respectively, and may be configured to perform a ground plane correction using this information. For example, while the vehicle102is moving, ground plane coefficients may be continuously changing depending on the inclination of the road surface, as well as pitching, rolling, and yaw angle of the vehicle102. To account for the changes in the ground plane coefficients, the ground plane estimator215may be configured to update the ground plane coefficients by measuring an actual inclination of the ground surface in real-time. To achieve this, the ground plane estimator215may be configured to determine ground plane coefficients using a RANSAC (Random Sample Consensus) algorithm for fitting to optimal coefficients (a, b, c, d) in equation (3) using the bottom four corners vertices of the 3D bounding box. As such, the ground plane coefficients may be refined to fit to the road surface in real-time, and in order to solve this problem, the ground plane estimator215may be configured to use the depth information from the depth network210A. In some implementations, the depth information from the depth network210A may be a normalized depth and the ground plane estimator215may be configured to convert the normalized depth to an actual distance. For example, the ground plane estimator215may convert the normalized depth into an actual distance based on the known distance beforehand, as illustrated in equation (4), shown above.

In further implementations, the ground plane estimator215may be configured to select a plurality of fixed points in a lower portion of the image of the input205that includes the road surface. After that, the ground plane estimator215may be configured to remove one or more points of the plurality of fixed points that are inappropriate for estimating the ground plane. To determine which points are inappropriate, the ground plane estimator215may determine which points are outside of the 3D bounding box for every frame. Using the information from the depth network210A, the ground plane estimator215may be configured to project the remaining points into a 3D world coordinate system and to execute the RANSAC algorithm to estimate the corrected ground plane coefficients. To update the ground plane, the ground plane estimator215may be configured to determine whether the update is possible by comparing the initial ground plane coefficients, ground plane coefficients in a previous frame, and currently estimated ground plane coefficients based on equation (5). When the update is determined to be possible, the ground plane estimator215may be configured to compute new coefficients of the ground plane based on equation (6) shown above. The term θ may be decided to ignore rapid changes by measurement noise, and may be decided by experimental way or scene. If θ is above a certain threshold value, the measurement noise may not be effectively removed. If θ is below another threshold value, the ground plane may be adjusted on small slope changes of the road. The threshold values may be empirically and/or experimentally determined.

In some aspects, there may be an assumption that the three bottom vertices is on the ground plane from the ground plane estimator215result. In some implementations, the 3D bounding box network210B may be further configured to determine the orientation230B of the surrounding vehicle in the 3D world coordinate system. For example, the orientation230B may be calculated by projecting the predicted 3D bounding box from the 3D bounding box network210B into the corrected ground plane from the ground plane estimator215using the depth estimation from the depth network210A. In some implementations, the flow network210C may determine the absolute velocity230C of the surrounding vehicle based on a difference between the absolute velocity of the autonomous vehicle and a relative velocity of the surrounding vehicle.

The method also includes determining the velocity of the autonomous vehicle based on an immovable object relative to the autonomous vehicle950. For example, the ego flow estimator225may be configured to determine the absolute velocity of the vehicle102based on the flow of immovable objects, such as the ground. For example, in order to estimate the velocity of the vehicle102, the ego flow estimator225may be configured to assume that a road surface as close as possible to the vehicle102is a fixed calculation area405, as illustrated inFIG. 4. The ego flow estimator225may be configured to determine a 2D flow vector (u, v) in this fixed calculation area405and to extract the 2D flow into a 3D flow (flowGx, flowGy, flowGz) using the projection matrix (1), (2) and the ground plane (3) received from the 3D bounding box network210B. Using this information, the ego flow estimator225may be configured to compute a ground speed VGat the fixed calculation area405using (7) shown above. In some aspects, an operation of the autonomous vehicle may be controlled based on at least one of the state information or the velocity of the autonomous vehicles.

The method includes determining the relative velocity of the surrounding vehicle based on the flow extraction estimator220. For example, the flow extraction estimator220may be configured to determine the relative velocity of surrounding vehicle based on projecting the flow into the plane of the detected 3D bonding box. The flow extraction estimator220may calculate the absolute velocity of surrounding vehicle using the relative velocity of the surrounding vehicle and the absolute velocity of the ego-vehicle calculated by the ego flow estimator225. To achieve this, the flow extraction estimator220may be configured to project the 2D flow of the surrounding vehicle to the 3D coordinate of the surrounding vehicle, rather than the ground plane of the vehicle. As discussed above, the 3D bounding box network210B may be configured to estimate the 3D bounding box of the surrounding vehicle with the 3D coordinates of each vertex. In some implementations, the flow extraction estimator220may convert an arbitrary 2D point on the 3D bounding box505into a 3D position and generate a new plane A passing through the arbitrary point and parallel to the ground plane GP, as illustrated inFIG. 5. Additionally, the flow extraction estimator220may be configured to convert the 2D flow into 3D flow based on the plane A corresponding to the surrounding vehicle. For example, the 2D flow at an arbitrary point may be projected on the plane A rather than the ground plane GP.

In some aspects of the present disclosure, the flow extraction estimator220may be configured to account for different shapes of vehicles. To achieve this, the flow extraction estimator220may be configured to calculate the 2D flow in a lower half of the 3D bounding box505. Moreover, the flow extraction estimator220may use two vertical planes of the 3D bounding box505near the vehicle102for 2D flow extraction as the two vertical planes may be visible from the vehicle102when there are no occlusions caused by other obstacles. In still further aspects, a 2D flow vector (u, v) in the 3D bounding box505may be projected on each plane based on the 2D coordinates, and then the flow extraction estimator220may compute the 3D flow (flowSx, flowSy, flowSz) and the relative velocity of the surrounding vehicle using equation (8) shown above.

In some examples, the flow extraction estimator220may be configured to convert the relative velocity to an absolute velocity. For example, in some implementations, flow extraction estimator220may calculate an absolute longitudinal velocity Va_szof the surrounding vehicle by subtracting the relative velocity of the surrounding vehicle Vr_szfrom the velocity VGzof the vehicle102based on equation 9 shown above.

In certain implementations, for the absolute lateral velocity, the flow extraction estimator220may be configured to recalculate a second lateral velocity V′Gxof the vehicle102near the surrounding vehicle from the based on calculated the lateral and longitudinal velocities VGz,VGzof the vehicle102. As illustrated inFIG. 6, a distance d0between RCego(e.g., a location on the vehicle102) and the camera center, e.g., a center of camera120) is constant for a given type of vehicle. As further illustrated inFIG. 6, a distance dGbetween the camera center and the center CGof the fixed ground plane, and the distance dSbetween the camera center and the center CSof the ground near the surrounding vehicle can be predicted using, for example, the ground plane estimator215and/or the depth network210A, as described herein. Using this information, the second lateral velocity V′Gxnear the surrounding vehicle may be calculated based on equation (10) shown above.

Using the second lateral velocity V′Gx, the flow extraction estimator220may be then calculate the absolute lateral velocity Va_sxof the surrounding vehicle based on equation (11) shown above. That is, the flow extraction estimator220may calculate the absolute longitudinal velocity Va_szand the absolute lateral velocity Va_sxof the surrounding vehicle, which may be output as the velocity230c.