alokanand002/dhem-entity-matching

DHEM Entity Matching

PyTorch pair scorers for matching noisy organization names against a supplied master catalog. Models return pairwise scores; use the catalog's canonical name as the resolved entity name and retain review handling for ambiguous matches.

Checkpoints

best.pt

• Catalog: data/master_entities.csv
• Training data: Curated master entities with synthetic name variants and approved aliases.
• Validation: Held-out normalized query groups; each master entity appears in both splits.
• Validation F1: 0.9907235621521336
• Parameters: 9,985
• Configuration: hidden dimension 128, dropout 0.15, learning rate 0.001, weight decay 0.0001, batch size 256, epochs 20
• SHA-256: 8d0afd0874ced1f1cf6ad61fbf7dc316aa35ec8ea15854ead96559f0a499d2e8

Architecture (input: 10 numeric pair features):

1. Linear (10 -> 128)
2. ReLU
3. LayerNorm (normalized_shape=128)
4. Dropout (p=0.15)
5. Linear (128 -> 64)
6. ReLU
7. Dropout (p=0.15)
8. Linear (64 -> 1)

Output: Squeeze final dimension to one binary match logit per query-candidate pair.

Training pairs: data/pairs/train.csv Validation pairs: data/pairs/val.csv

Pair examples

Use case	Query	Candidate	Pair score	Expected result
Positive, approved alias	JNJ	Johnson & Johnson	1.0000	MATCH
Positive, approved alias	3MLTD	3M	1.0000	MATCH
Negative, different entity	JNJ	3M	0.0020	NO_MATCH
Negative, different entity	3MLTD	Microsoft Corporation	0.0557	NO_MATCH

real_best.pt

• Catalog: data/pairs/real_master.csv
• Training data: Company records and source-backed aliases/former names from SEC and/or OpenAlex.
• Validation: Entity-aware split; validation entities are held out from training.
• Validation F1: 0.9296791815193741
• Parameters: 9,985
• Configuration: hidden dimension 128, dropout 0.15, learning rate 0.001, weight decay 0.0001, batch size 256, epochs 20
• SHA-256: d727e8e7b212684b34e06e3280d2c0735a23ed40a3a2804dbe9956be1fb7dc51

Architecture (input: 10 numeric pair features):

1. Linear (10 -> 128)
2. ReLU
3. LayerNorm (normalized_shape=128)
4. Dropout (p=0.15)
5. Linear (128 -> 64)
6. ReLU
7. Dropout (p=0.15)
8. Linear (64 -> 1)

Output: Squeeze final dimension to one binary match logit per query-candidate pair.

Training pairs: data/pairs/real_train.csv Validation pairs: data/pairs/real_val.csv

Pair examples

Use case	Query	Candidate	Pair score	Expected result
Positive, approved alias	NVIDIA CORP/CA	NVIDIA CORP	1.0000	MATCH
Negative, different entity	NVIDIA CORP/CA	Apple Inc.	0.0339	NO_MATCH

Input features

The model consumes ten deterministic pair features, in order: normalized equality, approved-alias equality, token Jaccard, token containment, character n-gram Jaccard, edit similarity, prefix similarity, acronym match, length ratio, and rule score. Full feature names are included in model_details.json.

Inference

Use each checkpoint with its corresponding master CSV:

python src/scripts/inference.py --checkpoint artifacts/best.pt --master data/master_entities.csv --query "JNJ"
python src/scripts/inference.py --checkpoint artifacts/real_best.pt --master data/pairs/real_master.csv --query "NVIDIA CORP/CA"

The best.pt checkpoint targets the curated catalog. real_best.pt targets the real SEC/OpenAlex catalog. Do not interchange their master files: an alias present in one catalog may be absent from the other. For hierarchical business decisions, use src/scripts/hierarchical_inference.py.

Limitations

Pair scores are model probabilities, not legal-identity guarantees. Validation scores are specific to each dataset and split and are not directly comparable estimates of performance on every production catalog. The real-data catalog can contain related but legally distinct organizations; use authoritative entity IDs and business rules, and route uncertain or related-entity cases for review.

Repository

Source code, dataset-generation scripts, and evaluation utilities: project repository.