Repository: huggingface/transformers
Pull request: [add-new-model-like] Robust search & proper outer '),' in tokenizer mapping

### What does this PR do?

This PR makes the `transformers-cli add-new-model-like` command usable again for any model whose tokenizer mapping is written on multiple lines (e.g. llama).

### Current failure modes

1. **`IndexError` while locating `TOKENIZER_MAPPING_NAMES`.**
   Inside function `transformers/src/transformers/commands/add_new_model_like.py/insert_tokenizer_in_auto_module()`. 
   The helper scans for the literal

   ```python
   "    TOKENIZER_MAPPING_NAMES = OrderedDict("
   ```

   (four leading spaces, no type annotation).
   Since in current version that line is un-indented **and** type-annotated:

   ```python
   TOKENIZER_MAPPING_NAMES = OrderedDict[str, tuple[Optional[str], Optional[str]]](
   ```

   The hard-coded `startswith()` never matches, the loop overruns `lines`, and the command aborts with

   ```
   IndexError: list index out of range
   ```

2. **Unbalanced parentheses once the above is patched.**
   When the mapping tokenizers in `transformers/src/transformers/models/auto/tokenization_auto.py` of which entry spans several lines, the script copies only the inner block ending in

   ```
               ),
   ```

   but forgets the outer

   ```
           ),
   ```

   line. Insertion borrows this outer `),` from the previous entry, leaving that entry syntactically broken and rendering `tokenization_auto.py` unimportable.

**Fix**

* Replace the fixed-width search with a regex tolerant of any indentation and optional type annotations:

  ```python
  pattern_tokenizer = re.compile(r"^\s*TOKENIZER_MAPPING_NAMES\s*=\s*OrderedDict\b")
  ```
* While copying a multi-line mapping block, keep collecting until the outer

  ```python
          ),
  ```

  line is also captured, ensuring the new block is fully closed before insertion.

No external dependencies are introduced-only the standard-library `re` module is used.

After the patch, running

```bash
transformers-cli add-new-model-like
```

completes without errors, and

```bash
python -m py_compile src/transformers/models/auto/tokenization_auto.py
```

succeeds.

I have not added a dedicated unit test; the change touches a dev-only CLI and has been verified manually. If desired, I can add a small test in `tests/commands/`. Also, I believe there's no documentation need to change based on this modification.

No issue number exists; the bug is reproducible via the steps above.

## Before submitting
- [ ] This PR fixes a typo or improves the docs
- [x] Did you read the contributor guideline?
- [ ] Was this discussed/approved via a GitHub issue or the forum?
- [ ] Did you make sure to update the documentation with your changes?
- [ ] Did you write any new necessary tests?

_No necessary documentation/testcases updates are required for this change._

## Who can review?
@ArthurZucker @gante


Changed files:
filename: src/transformers/commands/add_new_model_like.py
patch: @@ -12,6 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+# 1. Standard library
 import difflib
 import json
 import os
@@ -28,7 +29,12 @@
 
 from ..models import auto as auto_module
 from ..models.auto.configuration_auto import model_type_to_module_name
-from ..utils import is_flax_available, is_tf_available, is_torch_available, logging
+from ..utils import (
+    is_flax_available,
+    is_tf_available,
+    is_torch_available,
+    logging,
+)
 from . import BaseTransformersCLICommand
 from .add_fast_image_processor import add_fast_image_processor
 
@@ -1009,10 +1015,11 @@ def insert_tokenizer_in_auto_module(old_model_patterns: ModelPatterns, new_model
     with open(TRANSFORMERS_PATH / "models" / "auto" / "tokenization_auto.py", "r", encoding="utf-8") as f:
         content = f.read()
 
+    pattern_tokenizer = re.compile(r"^\s*TOKENIZER_MAPPING_NAMES\s*=\s*OrderedDict\b")
     lines = content.split("\n")
     idx = 0
     # First we get to the TOKENIZER_MAPPING_NAMES block.
-    while not lines[idx].startswith("    TOKENIZER_MAPPING_NAMES = OrderedDict("):
+    while not pattern_tokenizer.search(lines[idx]):
         idx += 1
     idx += 1
 
@@ -1024,9 +1031,12 @@ def insert_tokenizer_in_auto_module(old_model_patterns: ModelPatterns, new_model
         # Otherwise it takes several lines until we get to a "),"
         else:
             block = []
-            while not lines[idx].startswith("            ),"):
+            # should change to "        )," instead of "            ),"
+            while not lines[idx].startswith("        ),"):
                 block.append(lines[idx])
                 idx += 1
+            # if the lines[idx] does start with "        )," we still need it in our block
+            block.append(lines[idx])
             block = "\n".join(block)
         idx += 1