A system implemented on one or more data processing apparatuses, comprising: an interface configured to obtain, from one or more sensor subsystems, sensor data describing an environment of a vehicle, and to generate, using the sensor data, (i) one or more first neural network inputs for sensor measurements for a particular object in the environment, the one or more first neural network inputs comprising one or more patches of sensor data representing the particular object, the generating comprising extracting, from the sensor data, the one or more patches of sensor data comprising the particular object by cropping or isolating them from measurements for other portions of the environment represented in the sensor data, and (ii) a second neural network input representing sensor measurements for a wide-view representation of the environment, the sensor measurements for the wide-view representation comprising at least sensor measurements of a portion of the environment that are not included in the patches of sensor data, the generating comprising segmenting the wide-view representation into a plurality of regions of the environment, wherein the plurality of regions are defined by a grid with each region corresponding to a grid cell; a convolutional neural network configured to process the second neural network input to generate a context map, the context map being being an embedding or data structure that characterizes features of the environment based on the wide-view representation, and the context map including a plurality of feature vectors that each correspond to a different one of the plurality of regions of the environment; and an object classifier neural network configured to process the one or more first neural network inputs and a first feature vector of the plurality of feature vectors to generate a predicted classification for the particular object, wherein the first feature vector corresponds to a region in the plurality of regions that contains the particular object. an interface configured to obtain, from one or more sensor subsystems, sensor data describing an environment of a vehicle, and to generate, using the sensor data, (i) one or more first neural network inputs for sensor measurements for a particular object in the environment, the one or more first neural network inputs comprising one or more patches of sensor data representing the particular object, the generating comprising extracting, from the sensor data, the one or more patches of sensor data comprising the particular object by cropping or isolating them from measurements for other portions of the environment represented in the sensor data, and (ii) a second neural network input representing sensor measurements for a wide-view representation of the environment, the sensor measurements for the wide-view representation comprising at least sensor measurements of a portion of the environment that are not included in the patches of sensor data, the generating comprising segmenting the wide-view representation into a plurality of regions of the environment, wherein the plurality of regions are defined by a grid with each region corresponding to a grid cell; a convolutional neural network configured to process the second neural network input to generate a context map, the context map being being an embedding or data structure that characterizes features of the environment based on the wide-view representation, and the context map including a plurality of feature vectors that each correspond to a different one of the plurality of regions of the environment; and an object classifier neural network configured to process the one or more first neural network inputs and a first feature vector of the plurality of feature vectors to generate a predicted classification for the particular object, wherein the first feature vector corresponds to a region in the plurality of regions that contains the particular object.