The activation function should be used at the hidden layer. The output activation function should be either linear or softmax.