Amharic Skip-Gram Embeddings From Scratch
Model Description
A small Amharic skip-gram model implemented with Python lists, loops, and standard-library modules. It learns a center word's embedding by predicting nearby words with full softmax. The release contains the trained parameters in a custom JSON format and a standalone nearest-neighbor example, not a pretrained language model or a Transformers checkpoint.
Model Details
| Property | Value |
|---|---|
| Language | Amharic |
| Architecture | Skip-gram with full softmax |
| Vocabulary | 200 words |
| Embedding dimension | 12 |
| Input embeddings, E | 200 x 12 |
| Output weights, U | 12 x 200 |
| Trainable parameters | 4,800 = 2 x 200 x 12 |
| Implementation | Python standard library only |
| Saved fields | vocabulary, E, U, settings |
There are no pretrained embeddings, external numerical libraries, or automatic differentiation. Both matrices were learned from scratch. Nearest-neighbor queries use rows of E.
Training Data
The local, purpose-built corpus.txt contains 85 short Amharic sentences and 316 preprocessed tokens. Repeated contexts cover school, family, coffee and tea, farming, rain, and other everyday topics. The vocabulary has 200 distinct tokens. A window of two words on either side produces 754 directed center-context pairs. Each nonempty corpus line is one sentence; pairs never cross sentence boundaries.
This is a tiny educational corpus, not a representative sample of Amharic. The corpus itself is in the assignment source repository, not in this minimal model upload.
Preprocessing
Text is normalized to Unicode NFC. Unicode punctuation and symbols are replaced with spaces, then whitespace tokenization retains tokens containing at least one letter. Meaningful Amharic characters and word forms are preserved: there is no stemming, transliteration, or merging of spelling variants. The vocabulary is sorted deterministically, and each word's list position is its ID.
Architecture
Using row-vector notation, E is V x d and U is d x V. For center ID i and context target ID t:
h = E[i]
s = hU
m = max(s)
a[j] = exp(s[j] - m)
p[j] = a[j] / sum_r a[r]
L = (m - s[t]) + ln(sum_j a[j])
e = p - y # y[t] = 1; all other y[j] = 0
grad_U[k,j] = h[k] * e[j]
grad_h[k] = sum_j U[k,j] * e[j]
U = U - learning_rate * grad_U
E[i] = E[i] - learning_rate * grad_h
Both gradients are calculated using the old forward-pass parameter values before either matrix is updated. SGD updates all of U but only the selected row E[i]. Training pairs are shuffled each epoch. The reported dataset loss is J = (1/N) sum_n L_n, evaluated over every pair at the same fixed current weights without updates. The stable loss avoids taking the logarithm of an underflowed target probability.
Training Configuration
| Setting | Value |
|---|---|
| Context window | 2 |
| Embedding dimension | 12 |
| Learning rate | 0.05 |
| Epochs | 30 |
| Random seed | 2026 |
| Initialization | Independent uniform draws in [-0.5/d, 0.5/d] for E and U |
| Optimizer | Manually implemented SGD |
The compact dimension and corpus keep full-softmax training practical in pure Python. These settings train stably and produce some interpretable contextual neighbors, without implying broad semantic quality.
Evaluation and Verification
Results were reproduced with Python 3.11.9 by running both python main.py --checks and python main.py in the original assignment repository.
| Measurement or check | Verified result |
|---|---|
| Initial fixed-weight mean training loss | 5.298295 |
| Final fixed-weight mean training loss | 2.616722 |
| Relative loss decrease | 50.61% |
| Six mandatory checks | 6/6 PASS |
| Check tolerance | 0.00001 |
| Shift invariance (+100 to every score) | PASS |
| Central-difference gradient check | PASS; epsilon = 0.00001 |
| Maximum numerical-gradient error | 5.330e-12 |
| Fixed-weight evaluation without parameter updates | PASS |
| JSON save/reload | PASS |
| Real trained-model reload verification | PASS |
| Same-seed reproducibility | PASS |
| Unknown query and zero-norm cosine safety | PASS |
The mandatory checks cover sentence-local context pairs, forward scores, probabilities and loss, analytical gradients, one SGD update, and extreme scores [1000, 0]. Save/reload verification explicitly checks exact vocabulary equality, probabilities within tolerance, and identical nearest-neighbor order; it also checks settings. The gradient check samples entries from U and the selected E row. The same-seed check repeats a smaller training run on the real corpus and compares initialization, losses, parameters, and neighbors exactly.
These losses are training-objective measurements, not held-out semantic evaluation. No accuracy, F1, precision, recall, BLEU, or perplexity was measured.
Example Nearest Neighbors
These are the top three other vocabulary words by manually computed cosine similarity between rows of the learned E matrix. They are contextual or grammatical neighbors, not necessarily synonyms.
| Query | Top three neighbors (cosine similarity) |
|---|---|
| α°ααͺ | α α¨αα (0.7980), αα ααα (0.7303), ααα΅α (0.6780) |
| α α΅α°ααͺ | α«α΅α¨α³α (0.8595), αα ααα (0.7946), αααα¨α³α (0.7866) |
| α‘α | α΅ααα£αα½ (0.7295), α»α (0.6580), α΅αααα½ (0.6580) |
| αα α¬ | α΅αα΄ (0.8106), α ααα₯ (0.7330), αα α«α (0.6733) |
| ααα₯ | α«ααα£α (0.8252), αα¨α₯α£α (0.8071), αα¬α΅ (0.7792) |
Controlled Context-Window Experiment
Only the context window changed. The corpus, preprocessing, vocabulary, dimension, learning rate, epochs, and seed stayed fixed; each run started from the same seeded initialization.
| Context window | Directed training pairs | Final mean loss |
|---|---|---|
| 2 (published model) | 754 | 2.616722 |
| 1 (comparison) | 462 | 3.358973 |
For window 1, three fixed queries produced:
| Query | Top three neighbors (cosine similarity) |
|---|---|
| α°ααͺ | αα ααα (0.8754), α«α αα (0.8073), α α΅α°ααͺ (0.7865) |
| α‘α | α΅αααα½ (0.9380), α»α (0.8088), α«α³α΅αα (0.7778) |
| αα α¬ | ααα«α (0.9162), α ααα₯ (0.8929), αα°α α΅α£α (0.8676) |
Compare these with the window-2 rows above. The losses are averages over different training-pair datasets, so the lower value does not prove that one context window is universally better. Only the window-2 model is included in model.json.
Intended Use
Classroom demonstrations of skip-gram, stable softmax, manual backpropagation, SGD, cosine similarity, and reproducible training. Suitable for inspecting a small Amharic embedding space and explaining the mathematics line by line.
Limitations
- The training corpus is very small, and the vocabulary is limited to 200 words.
- Amharic morphology is not explicitly modeled; related inflected forms remain separate tokens.
- Unknown words have no embedding; zero-norm cosine is safely returned as 0.0.
- Nearest neighbors may reflect contextual or grammatical similarity rather than synonyms.
- Lower training loss does not establish general semantic quality.
- This is an educational, small-scale model, not production-scale embeddings.
- The custom JSON format requires explicit loading; it does not provide a hosted inference widget or a Transformers pipeline.
Usage
Download model.json and usage.py into the same directory. No packages need to be installed.
python usage.py
python usage.py "α‘α"
python usage.py "α°ααͺ"
The default query is α‘α. The script prints three neighbors, excludes the query itself, and reports unknown words without crashing. The JSON contains vocabulary, E, U, and settings; E[vocabulary.index(word)] retrieves a word's embedding. Tied similarities are ordered by word for deterministic results.
Reproducibility
The verified assignment source is amharic-word-embeddings-from-scratch, revision 7a8b2045610e603767e81efaa90027176999a366. That GitHub repository is private, so access is required to obtain the training code and corpus. This model package supports standalone loading and querying, but does not by itself include everything needed to retrain.
With access to the source, run python main.py --checks, then python main.py. The normal run prints every epoch's fixed-weight loss, saves the final JSON, verifies reloading, and repeats the one-variable context-window experiment. Keep the sorted vocabulary, corpus sentence order, seed, and training configuration unchanged to reproduce the results. Small floating-point differences may occur across Python versions or platforms.
License
The published model parameters (model.json), standalone query code (usage.py), and this model card are released under the MIT License.
This license does not assert any rights or license for the underlying training corpus. The corpus is not uploaded here; permission to redistribute it has not been established for this release. The original private assignment repository is separate from this model release.