An RNN consists of a single hidden layer that maintains a hidden state or memory. This hidden state is updated at each time step and influences the prediction made at that step. It allows the network to handle sequential data efficiently.