Before revision: We introduce a very deep and light-weight transformer, DeLighT, that delivers similar or better performance than  transformer-based models with significantly fewer parameters.
After revision: We introduce a very deep and light-weight transformer, DeLighT, that delivers similar or better performance than standard transformer-based models with significantly fewer parameters.