Before revision: We introduce a very deep and light-weight transformer, DeLighT, that delivers similar or better performance than  transformer-based models with significantly fewer parameters.
After revision: We introduce a deep and light-weight transformer, DeLighT, that delivers similar or better performance than transformer-based models with significantly fewer parameters.