Fix Inconsistent Token Decoding in Qwen2 Tokenizer #35641
Author: sambhavnoobcoder
Pull request: yes

## Issue Description
The Qwen2.5-7B-Instruct tokenizer exhibits inconsistent behavior when decoding special characters. Specifically, when using token ID 124 as an example:
- `convert_ids_to_tokens()` returns 'À' (correct)
- `decode()` returns '�' (replacement character, incorrect)

This inconsistency affects any application that relies on consistent token decoding, particularly when dealing with non-ASCII characters or special tokens.

Fixes #35641 

## Analysis

### Root Cause
1. **Encoding Mismatch**: 
   - The tokenizer stores special characters using Latin-1 encoding in its vocabulary
   - The default decoding process assumes UTF-8 encoding
   - This mismatch causes special characters to be incorrectly decoded as replacement characters

2. **Tokenizer JSON Format**:
   - The tokenizer initialization wasn't properly handling the SentencePiece-style JSON format
   - This led to incorrect vocabulary loading and token mapping

3. **Inconsistent Handling**:
   - `convert_ids_to_tokens()` directly maps IDs to tokens, preserving special characters
   - `decode()` applies additional processing that incorrectly handles byte-level encoding

## Solution

### Approach
1. **Character Encoding**:
   - Implement proper encoding/decoding chain:
     1. Encode tokens as Latin-1 to preserve byte values
     2. Try UTF-8 decoding first (for normal characters)
     3. Fall back to Latin-1 decoding for special characters

2. **Tokenizer Initialization**:
   - Add structured parsing of tokenizer.json:
     1. Load special tokens first
     2. Load regular vocabulary
     3. Maintain proper token-to-ID mappings

3. **Consistent Processing**:
   - Special handling for single tokens to ensure consistency
   - Maintain compatibility with batch processing
   - Preserve error handling configuration

### Implementation Details
1. **Vocabulary Loading**:
   - Parse both special tokens and regular vocabulary
   - Handle SentencePiece-style JSON format
   - Maintain proper token ordering

2. **Character Encoding**:
   - Implement Latin-1/UTF-8 encoding chain
   - Add fallback mechanisms for special characters
   - Preserve error handling configuration

3. **Method Enhancement**:
   - Modify `convert_tokens_to_string()`
   - Enhance `decode()` method

## Testing

### Test Code

```
import os
import sys
import json

# Add the src directory to the Python path
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "src")))

from transformers.models.qwen2.tokenization_qwen2 import Qwen2Tokenizer

# Get the cache directory where the tokenizer files are downloaded
cache_dir = os.path.expanduser("~/.cache/huggingface/hub/models--Qwen--Qwen2.5-7B-Instruct/snapshots/")
# Find the most recent snapshot
snapshot_dir = max(os.path.join(cache_dir, d) for d in os.listdir(cache_dir))
vocab_file = os.path.join(snapshot_dir, "tokenizer.json")
merges_file = os.path.join(snapshot_dir, "merges.txt")

# Create tokenizer instance directly using our local implementation
tokenizer = Qwen2Tokenizer(vocab_file=vocab_file, merges_file=merges_file)

token_id = 124
print([tokenizer.convert_ids_to_tokens([token_id])[0], tokenizer.decode(token_id)])
```

### Results

#### Before Fix:
<img width="83" alt="Screenshot 2025-01-13 at 3 00 47 AM" src="https://github.com/user-attachments/assets/2aa90305-e729-4998-b40f-3b5651556ec4" />

#### After Fix:
<img width="75" alt="Screenshot 2025-01-13 at 3 01 20 AM" src="https://github.com/user-attachments/assets/f6b8705d-51d9-4524-b470-dbae17734182" />

## Related Issues
- #35641 : Original issue report

Comments:
cc @ArthurZucker for tokenizer issues 

EDIT: actually, I think this was already addressed in #35641, never mind

okay , thanks . i'll close this PR then