Before revision:  In this paper, we  show how a standard transformer can be morphed into a hierarchical one by using specially designed attention masks and positional embeddings. Our experiments show strong improvements in context-to-response generation performance for task-oriented dialog system over the current state-of-the-art approaches .
After revision:  In this paper, we show how a standard transformer can be morphed into a hierarchical one by using specially designed attention masks and positional encodings. We demonstrate that Hierarchical Encoding helps achieve better natural language understanding of the contexts in transformer-based models for task-oriented dialog system over the current state-of-the-art approaches .