DeepChem

/

ChemBERTa-77M-MLM

like 26

Follow

DeepChem 119

Hi!

I think the tokenizer for this model and other models that use atom-wise tokenization is faulty. Here, I show one of the example.

Input: "CCCCCCCCCCCCCCCCCCP+([O-])O.CCCCCCCCCCCCCCCCCCP+([O-])[O-].[La+3]"
Expected Tokens (Length = 55): ['C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', '[P+]', '(', '[O-]', ')', '(', '[O-]', ')', 'O', '.', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', '[P+]', '(', '[O-]', ')', '(', '[O-]', ')', '[O-]', '.', '[La+3]']
Observed Tokens (Length = 60): ['C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'P', '(', 'O', '-', ')', '(', 'O', '-', ')', 'O', '.', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'P', '(', 'O', '-', ')', '(', 'O', '-', ')', 'O', '-', '.', '3']

It seems that the tokenizer no longer respect the square brackets and +/-. I believe that a different version of the library that may interpret the regex differently, since I saw this across different models.