An apparatus comprising a processor configured to implement a convolutional neural network for determining a mode decision for encoding a block in video coding, the convolutional neural network comprising: feature extraction layers for extracting features of the block for determining the mode decision, wherein a non-overlapping convolution operation is performed on input at at least one of the feature extraction layers by setting a stride value equal to a kernel size, the mode decision comprises a block partitioning of the block, the block has a N×N size, and a smallest partition output for the block has a S×S size; and multiple classifiers for processing the features of the block, wherein: each classifier comprises classification layers, each classification layer of the classification layers for receiving respective feature maps having a respective feature dimension, each classifier is configured to infer partition decisions for sub-blocks of size (αS)×(αS) of the block, wherein α is a power of 2 and α=2, ..., N/S, by: applying, at some of successive classification layers of the classification layers, a kernel of size 1×1 to reduce the respective feature dimension; and outputting, by a last layer of the classification layers, an output corresponding to a N/(αS)×N/(αS)xl output map, the output map indicating one or more partition decisions for the block. feature extraction layers for extracting features of the block for determining the mode decision, wherein a non-overlapping convolution operation is performed on input at at least one of the feature extraction layers by setting a stride value equal to a kernel size, the mode decision comprises a block partitioning of the block, the block has a N×N size, and a smallest partition output for the block has a S×S size; and multiple classifiers for processing the features of the block, wherein: each classifier comprises classification layers, each classification layer of the classification layers for receiving respective feature maps having a respective feature dimension, each classifier is configured to infer partition decisions for sub-blocks of size (αS)×(αS) of the block, wherein α is a power of 2 and α=2, ..., N/S, by: applying, at some of successive classification layers of the classification layers, a kernel of size 1×1 to reduce the respective feature dimension; and outputting, by a last layer of the classification layers, an output corresponding to a N/(αS)×N/(αS)xl output map, the output map indicating one or more partition decisions for the block. each classifier comprises classification layers, each classification layer of the classification layers for receiving respective feature maps having a respective feature dimension, each classifier is configured to infer partition decisions for sub-blocks of size (αS)×(αS) of the block, wherein α is a power of 2 and α=2, ..., N/S, by: applying, at some of successive classification layers of the classification layers, a kernel of size 1×1 to reduce the respective feature dimension; and outputting, by a last layer of the classification layers, an output corresponding to a N/(αS)×N/(αS)xl output map, the output map indicating one or more partition decisions for the block. applying, at some of successive classification layers of the classification layers, a kernel of size 1×1 to reduce the respective feature dimension; and outputting, by a last layer of the classification layers, an output corresponding to a N/(αS)×N/(αS)xl output map, the output map indicating one or more partition decisions for the block. <EOS>