Break out of layer construction if there is not enough data