A computer-implemented method for making a decision in operating an autonomous driving vehicle, ADV, using machine learning, comprising: making one or more yield/overtake decisions with respect to one or more objects in the ADV's surrounding environment using a data processing architecture comprising at least a first, a second, and a third neural networks, the first, the second, and the third neural networks having been trained with a training data set; and generating driving signals based at least in part on the yield/overtake decisions to control operations of the ADV; wherein the first neural network is a multilayer perceptron, MLP, wherein the second neural network is a convolutional neural network, CNN, and wherein the third neural network is a fully-connected network; wherein the first neural network receives historical features of the one or more objects from one or more previous planning cycles as inputs, and generates extracted historical features of the one or more objects as outputs; wherein the second neural network receives the extracted historical features of the one or more objects and map information as inputs, and generates data encoding both the extracted historical features of the one or more objects and the map information as outputs; and wherein the third neural network receives the encoded data and historical features of the ADV as inputs, and generates the one or more yield/overtake decisions comprising decisions with respect to each of the one or more objects as outputs; wherein before the extracted historical features of the one or more objects and the map information are output to the second neural network, the extracted historical features of the one or more objects and the map information are labeled with associated block information based on a grid subdivision of a rectangular perception area of the ADV, the grid subdivision comprising subdividing the rectangular perception area of the ADV into a plurality of uniformly sized rectangular blocks based on a grid. making one or more yield/overtake decisions with respect to one or more objects in the ADV's surrounding environment using a data processing architecture comprising at least a first, a second, and a third neural networks, the first, the second, and the third neural networks having been trained with a training data set; and generating driving signals based at least in part on the yield/overtake decisions to control operations of the ADV; wherein the first neural network is a multilayer perceptron, MLP, wherein the second neural network is a convolutional neural network, CNN, and wherein the third neural network is a fully-connected network; wherein the first neural network receives historical features of the one or more objects from one or more previous planning cycles as inputs, and generates extracted historical features of the one or more objects as outputs; wherein the second neural network receives the extracted historical features of the one or more objects and map information as inputs, and generates data encoding both the extracted historical features of the one or more objects and the map information as outputs; and wherein the third neural network receives the encoded data and historical features of the ADV as inputs, and generates the one or more yield/overtake decisions comprising decisions with respect to each of the one or more objects as outputs; wherein before the extracted historical features of the one or more objects and the map information are output to the second neural network, the extracted historical features of the one or more objects and the map information are labeled with associated block information based on a grid subdivision of a rectangular perception area of the ADV, the grid subdivision comprising subdividing the rectangular perception area of the ADV into a plurality of uniformly sized rectangular blocks based on a grid.