Create README.md

README.md

@@ -0,0 +1,144 @@
+---
+license: apache-2.0
+language:
+- es
+---
+# PL-BERT-wordpiece-es-
+
+## Overview
+
+<details>
+<summary>Click to expand</summary>
+
+- **Model type:** Phoneme-level Language Model (PL-BERT)  
+- **Architecture:** AlBERT-base (12 layers, 768 hidden units, 12 attention heads)  
+- **Language:** Spanish  
+- **License:** Apache 2.0  
+- **Data:** Crowdsourced phonemized Spanish speech text  
+
+</details>
+
+---
+
+## Model description
+
+**PL-BERT-wordpiece-es** is a phoneme-level masked language model trained on Spanish text with diverse regional accents. It is based on the [PL-BERT architecture](https://github.com/yl4579/PL-BERT), which learns phoneme representations via a BERT-style masked language modeling objective.
+
+This model is designed to support **phoneme-based text-to-speech (TTS) systems**, including but not limited to [StyleTTS2](https://github.com/yl4579/StyleTTS2). Thanks to its Spanish-specific phoneme vocabulary and contextual embedding capabilities, it can serve as a phoneme encoder for any TTS architecture requiring phoneme-level features.
+
+Features of our PL-BERT:
+- It is trained **exclusively on Spanish** phonemized text.
+- It uses a reduced **phoneme vocabulary of 178 tokens**.
+- It uses wordpiece tokenizer.
+- It includes a custom `token_maps.pkl` and adapted `util.py`.
+
+---
+
+## Intended uses and limitations
+
+### Intended uses
+
+- Integration into phoneme-based TTS pipelines such as StyleTTS2, Matxa-TTS, or custom diffusion-based synthesizers.
+- Accent-aware synthesis and phoneme embedding extraction for Spanish.
+
+
+### Limitations
+
+- Not designed for general NLP tasks like classification or sentiment analysis.
+- Only supports Spanish phoneme tokens.
+- Some accents may be underrepresented in the training data.
+
+---
+
+## How to use (with StyleTTS2)
+
+Here is an example of how to use this model within the StyleTTS2 framework:
+
+1. Clone the StyleTTS2 repository: https://github.com/yl4579/StyleTTS2
+2. Inside the `Utils` directory, create a new folder, for example: `PLBERT_es`.
+3. Copy the following files into that folder:
+   - `config.yml` (training configuration)
+   - `step_1000000.t7` (trained checkpoint)
+   - `token_maps.pkl` (phoneme to ID mapping)
+   - `util.py` (modified to fix position ID loading)
+
+4. In your StyleTTS2 configuration file, update the `PLBERT_dir` entry to:
+
+   `PLBERT_dir: Utils/PLBERT_es`
+
+5. Update the import statement in your code to:
+
+   `from Utils.PLBERT_es.util import load_plbert`
+
+6. Use `espeak-ng` with the language code `es-419` to phonemize your Spanish text files for training and validation.
+
+Note: Although this example uses StyleTTS2, the model is compatible with other TTS architectures that operate on phoneme sequences. You can use the contextualized phoneme embeddings from PL-BERT in any compatible synthesis system.
+
+---
+
+## Training
+
+### Training data
+
+The model was trained on a Spanish corpus phonemized using espeak-ng. It uses a consistent phoneme token set with boundary markers and masking tokens.
+
+Tokenizer: custom (split using whitespaces)  
+Phoneme masking strategy: word-level and phoneme-level masking and replacement  
+Training steps: 1,000,000  
+Precision: Mixed (fp16)
+
+### Training configuration
+
+Model parameters:
+
+- Vocabulary size: 178  
+- Hidden size: 768  
+- Attention heads: 12  
+- Intermediate size: 2048  
+- Number of layers: 12  
+- Max position embeddings: 512  
+- Dropout: 0.1  
+
+Other parameters:
+
+- Batch size: 8  
+- Max mel length: 512  
+- Word mask probability: 0.15  
+- Phoneme mask probability: 0.1  
+- Replacement probability: 0.2  
+- Token separator: space  
+- Token mask: M  
+- Word separator ID: 102
+
+---
+
+
+## Evaluation
+
+The model has not been benchmarked via perplexity or extrinsic evaluation, but has been successfully integrated into TTS pipelines such as StyleTTS2, where it enables the synthesis of Spanish.
+
+---
+
+## Additional information
+
+### Contact
+
+For questions or feedback, please contact:  
+rodolfo.zevallos@bsc.es
+
+### License
+
+Distributed under the Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0
+
+### Citation
+
+Citation coming soon. Please cite the Hugging Face model card once published.
+
+### Disclaimer
+
+<details>
+<summary>Click to expand</summary>
+
+This model is released for research and educational use. It may exhibit biases or limitations based on training data characteristics. Users are responsible for ensuring appropriate use in deployed systems and for complying with all applicable regulations.
+
+</details>