In layerwise training, do not duplicate the final output layer, but instead train the true final output layer in the last pass.