Update README.md

README.md

@@ -1,86 +1,169 @@
 ---
-library_name: keras
+language:
+  - id
+  - ms
+license: apache-2.0
+tags:
+  - g2p
+  - fill-mask
+inference: false
 ---
 
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- optimizer: {'name': 'Adam', 'learning_rate': 0.001, 'decay': 0.0, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-07, 'amsgrad': False}
-- training_precision: float32
-
- ## Training Metrics
-
-| Epochs | Train Loss |
- |--- |--- |
-| 1| 0.43| 
-| 2| 0.386| 
-| 3| 0.356| 
-| 4| 0.311| 
-| 5| 0.303| 
-| 6| 0.296| 
-| 7| 0.293| 
-| 8| 0.286| 
-| 9| 0.28| 
-| 10| 0.272| 
-| 11| 0.263| 
-| 12| 0.261| 
-| 13| 0.256| 
-| 14| 0.252| 
-| 15| 0.255| 
-| 16| 0.246| 
-| 17| 0.247| 
-| 18| 0.248| 
-| 19| 0.244| 
-| 20| 0.24| 
-| 21| 0.24| 
-| 22| 0.234| 
-| 23| 0.234| 
-| 24| 0.229| 
-| 25| 0.231| 
-| 26| 0.23| 
-| 27| 0.236| 
-| 28| 0.227| 
-| 29| 0.224| 
-| 30| 0.221| 
-| 31| 0.221| 
-| 32| 0.221| 
-| 33| 0.222| 
-| 34| 0.224| 
-| 35| 0.222| 
-| 36| 0.215| 
-| 37| 0.214| 
-| 38| 0.208| 
-| 39| 0.208| 
-| 40| 0.205| 
-| 41| 0.206| 
-| 42| 0.204| 
-| 43| 0.204| 
-| 44| 0.201| 
-| 45| 0.2| 
-| 46| 0.199| 
-| 47| 0.197| 
-| 48| 0.195| 
-| 49| 0.199| 
-| 50| 0.195| 
- ## Model Plot
+# ID G2P BERT
+
+ID G2P BERT is a phoneme de-masking model based on the [BERT](https://arxiv.org/abs/1810.04805) architecture. This model was trained from scratch on a modified [Malay/Indonesian lexicon](https://huggingface.co/datasets/bookbot/id_word2phoneme).
+
+This model was trained using the [Keras](https://keras.io/) framework. All training was done on Google Colaboratory. We adapted the [BERT Masked Language Modeling training script](https://keras.io/examples/nlp/masked_language_modeling) provided by the official Keras Code Example.
+
+## Model
+
+| Model         | #params | Arch. | Training/Validation data |
+| ------------- | ------- | ----- | ------------------------ |
+| `id-g2p-bert` | 200K    | BERT  | Malay/Indonesian Lexicon |
+
+![](./model.png)
+
+## Training Procedure
+
+<details>
+  <summary>Model Config</summary>
+
+    vocab_size: 32
+    max_len: 32
+    embed_dim: 128
+    num_attention_head: 2
+    feed_forward_dim: 128
+    num_layers: 2
+
+</details>
+
+<details>
+  <summary>Training Setting</summary>
+
+    batch_size: 32
+    optimizer: "adam"
+    learning_rate: 0.001
+    epochs: 100
+
+</details>
+
+## How to Use
 
 <details>
-<summary>View Model Plot</summary>
+  <summary>Tokenizers</summary>
+
+    id2token = {
+        0: '',
+        1: '[UNK]',
+        2: 'a',
+        3: 'n',
+        4: 'ə',
+        5: 'i',
+        6: 'r',
+        7: 'k',
+        8: 'm',
+        9: 't',
+        10: 'u',
+        11: 'g',
+        12: 's',
+        13: 'b',
+        14: 'p',
+        15: 'l',
+        16: 'd',
+        17: 'o',
+        18: 'e',
+        19: 'h',
+        20: 'c',
+        21: 'y',
+        22: 'j',
+        23: 'w',
+        24: 'f',
+        25: 'v',
+        26: '-',
+        27: 'z',
+        28: "'",
+        29: 'q',
+        30: '[mask]'
+    }
+
+    token2id = {
+        '': 0,
+        "'": 28,
+        '-': 26,
+        '[UNK]': 1,
+        '[mask]': 30,
+        'a': 2,
+        'b': 13,
+        'c': 20,
+        'd': 16,
+        'e': 18,
+        'f': 24,
+        'g': 11,
+        'h': 19,
+        'i': 5,
+        'j': 22,
+        'k': 7,
+        'l': 15,
+        'm': 8,
+        'n': 3,
+        'o': 17,
+        'p': 14,
+        'q': 29,
+        'r': 6,
+        's': 12,
+        't': 9,
+        'u': 10,
+        'v': 25,
+        'w': 23,
+        'y': 21,
+        'z': 27,
+        'ə': 4
+    }
+
+</details>
+
+```py
+import keras
+import tensorflow as tf
+import numpy as np
+
+mlm_model = keras.models.load_model(
+    "bert_mlm.h5", custom_objects={"MaskedLanguageModel": MaskedLanguageModel}
+)
+
+MAX_LEN = 32
+
+def inference(sequence):
+    sequence = " ".join([c if c != "e" else "[mask]" for c in sequence])
+    tokens = [token2id[c] for c in sequence.split()]
+    pad = [token2id[""] for _ in range(MAX_LEN - len(tokens))]
+
+    tokens = tokens + pad
+    input_ids = tf.convert_to_tensor(np.array([tokens]))
+    prediction = mlm_model.predict(input_ids)
+
+    # find masked idx token
+    masked_index = np.where(input_ids == mask_token_id)
+    masked_index = masked_index[1]
+
+    # get prediction at those masked index only
+    mask_prediction = prediction[0][masked_index]
+    predicted_ids = np.argmax(mask_prediction, axis=1)
+
+    # replace mask with predicted token
+    for i, idx in enumerate(masked_index):
+        tokens[idx] = predicted_ids[i]
+
+    return "".join([id2token[t] for t in tokens if t != 0])
+
+inference("mengembangkannya")
+```
+
+## Authors
+
+ID G2P BERT was trained and evaluated by [Ananto Joyoadikusumo](https://anantoj.github.io/), [Steven Limcorn](https://stevenlimcorn.github.io/), [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory.
 
-![Model Image](./model.png)
+## Framework versions
 
-</details>
+- Keras 2.8.0
+- TensorFlow 2.8.0