Before revision:  Our scheme improves the computational cost and number of parameters in an attention layer and can be used as a drop-in replacement in any transformer architecture.
After revision:  Our scheme decreases the number of parameters in an attention layer and can be used as a drop-in replacement in any transformer architecture.