While the talking-heads approach used in GvT has shown promising results in enhancing the expressive power of attention heads, there are potential limitations and drawbacks that need to be addressed. One limitation is the increased computational complexity introduced by the additional linear projections and sparse selection operations. This can lead to longer training times and higher resource requirements. To address this, future work could focus on optimizing the implementation of talking-heads to reduce computational overhead without compromising performance. Another drawback is the potential for information loss during the sparse selection process, which may impact the model's ability to capture fine-grained details. Future research could explore alternative methods for selecting and combining attention heads to mitigate this issue and improve the overall performance of the model.