text: 
  
  TITLE
  Strange implementation of `convert_tokens_to_string` in albert tokenizer.
  
  COMMENTS
  6
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  Hi,
  
  the albert tokenizer implements the `convert_tokens_to_string` function:
  
  https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/albert/tokenization_albert.py#L222-L223
  
  While the deberta v2 and some other tokenizer just delegate this to the sentencepiece tokenizer:
  
  https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/deberta_v2/tokenization_deberta_v2.py#L146
  
  IMO it would be better to always delegate to the sentencepiece tokenizer. What do you think?
  
  ## PS:
  Some more examples here 
  
  https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/barthez/tokenization_barthez.py#L251-L252
  
  https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/camembert/tokenization_camembert.py#L251-L252
  
  https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/m2m_100/tokenization_m2m_100.py#L187-L188
  
  https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/mbart/tokenization_mbart50.py#L208-L209
  
  https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/speech_to_text/tokenization_speech_to_text.py#L169-L173
  
  https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/xlm_prophetnet/tokenization_xlm_prophetnet.py#L264-L265