A method for classifying physical objects comprising: controlling, by a processor, one or more depth cameras to capture a plurality of depth images of a query object; controlling, by the processor, one or more color cameras to capture a plurality of color images of the query object; computing, by the processor, a three-dimensional model of the query object using the depth images; wherein the 3D model comprises a 3D mesh model computed from the depth images; combining, by the processor, the color images with the 3D mesh model; computing, by the processor, a descriptor from the 3D mesh model and the color images, computing the descriptor comprises: rendering, by the processor, a plurality of standardized two-dimensional views of the 3D mesh model of the query object; and supplying, by the processor, the plurality of 2D views of the query object to a convolutional stage of a neural network to extract the descriptor; the descriptor comprising: a multi-dimensional shape descriptor space representation of a 3D shape of the query object; a multi-dimensional color descriptor space representation of a texture of the query object; and a one-dimensional size descriptor space representation of a size of the query object; supplying, by the processor, the descriptor to a classifier to compute a classification of the query object; outputting the classification of the query object controlling, by a processor, one or more depth cameras to capture a plurality of depth images of a query object; controlling, by the processor, one or more color cameras to capture a plurality of color images of the query object; computing, by the processor, a three-dimensional model of the query object using the depth images; wherein the 3D model comprises a 3D mesh model computed from the depth images; combining, by the processor, the color images with the 3D mesh model; computing, by the processor, a descriptor from the 3D mesh model and the color images, computing the descriptor comprises: rendering, by the processor, a plurality of standardized two-dimensional views of the 3D mesh model of the query object; and supplying, by the processor, the plurality of 2D views of the query object to a convolutional stage of a neural network to extract the descriptor; rendering, by the processor, a plurality of standardized two-dimensional views of the 3D mesh model of the query object; and supplying, by the processor, the plurality of 2D views of the query object to a convolutional stage of a neural network to extract the descriptor; the descriptor comprising: a multi-dimensional shape descriptor space representation of a 3D shape of the query object; a multi-dimensional color descriptor space representation of a texture of the query object; and a one-dimensional size descriptor space representation of a size of the query object; supplying, by the processor, the descriptor to a classifier to compute a classification of the query object; outputting the classification of the query object a multi-dimensional shape descriptor space representation of a 3D shape of the query object; a multi-dimensional color descriptor space representation of a texture of the query object; and a one-dimensional size descriptor space representation of a size of the query object; supplying, by the processor, the descriptor to a classifier to compute a classification of the query object; outputting the classification of the query object