A computer-implemented method for representing a surrounding environment of an autonomous driving vehicle, ADV, the method comprising: for a driving cycle, extracting a plurality of features from a portion of high definition, HD,map including a plurality of map layers using a convolution neural network, wherein the portion of the HD map is within an angle of view of the ADV, the angle of view being based on an angle corresponding to a heading of the ADV, and the plurality of map layers comprise a lane feature layer, a traffic sign layer, a static object layer and a general map information layer; concatenating the plurality of features extracted from the HD map with a plurality of handcrafted features of a target object in the surrounding environment, the target object being within the angle of view of the ADV at the driving cycle, the handcrafted features including properties of the target object that are relevant to a prediction of a behavior of the target object and derived from data obtained by one or more sensors, the properties of the target object including at least one of GPS coordinates of the target object, a width of the target object, a length of the target object, a vehicle type, or a speed of the target object; and predicting one or more behaviors of the target object by providing the concatenated features to one or more fully connected layers of the convolution neural network wherein extracting the plurality of features from the HD map includes: extracting the plurality of layers from the HD map, each of the layers being a geographic information system containing groups of point, line, or area features representing a class or type of real world entities; converting the plurality of layers into a red, green and blue, RGB, representation, the RGB representation being an m-by-n-by-3 data array that defines red, green and blue color components for each individual pixel; and extracting the plurality of features from the RGB representation using one or more convolution layers of the convolution neural network. for a driving cycle, extracting a plurality of features from a portion of high definition, HD,map including a plurality of map layers using a convolution neural network, wherein the portion of the HD map is within an angle of view of the ADV, the angle of view being based on an angle corresponding to a heading of the ADV, and the plurality of map layers comprise a lane feature layer, a traffic sign layer, a static object layer and a general map information layer; concatenating the plurality of features extracted from the HD map with a plurality of handcrafted features of a target object in the surrounding environment, the target object being within the angle of view of the ADV at the driving cycle, the handcrafted features including properties of the target object that are relevant to a prediction of a behavior of the target object and derived from data obtained by one or more sensors, the properties of the target object including at least one of GPS coordinates of the target object, a width of the target object, a length of the target object, a vehicle type, or a speed of the target object; and predicting one or more behaviors of the target object by providing the concatenated features to one or more fully connected layers of the convolution neural network wherein extracting the plurality of features from the HD map includes: extracting the plurality of layers from the HD map, each of the layers being a geographic information system containing groups of point, line, or area features representing a class or type of real world entities; converting the plurality of layers into a red, green and blue, RGB, representation, the RGB representation being an m-by-n-by-3 data array that defines red, green and blue color components for each individual pixel; and extracting the plurality of features from the RGB representation using one or more convolution layers of the convolution neural network. extracting the plurality of layers from the HD map, each of the layers being a geographic information system containing groups of point, line, or area features representing a class or type of real world entities; converting the plurality of layers into a red, green and blue, RGB, representation, the RGB representation being an m-by-n-by-3 data array that defines red, green and blue color components for each individual pixel; and extracting the plurality of features from the RGB representation using one or more convolution layers of the convolution neural network.