category-of-needs-classification-mpnet-v2
A SetFit multi-label classifier fine-tuned on paraphrase-multilingual-mpnet-base-v2 to identify the categories of human need expressed in natural-language search queries on the Connect211 social-services search platform.
Other models in this family
Three backbone variants were trained and evaluated side-by-side. Choose based on your accuracy vs. latency vs. memory trade-offs:
| Model | Gold Hit Rate | Top-1 Exact | Recall@5 | p50 Latency | Memory |
|---|---|---|---|---|---|
| mpnet-v2 ← this model | 74.4% | 28.2% | 63.8% | 28 ms | 838 MB |
| minilm-v2 | 72.5% | 17.9% | 60.4% | 21 ms | 789 MB |
| bge-m3-v2 | 63.1% | 15.4% | 58.9% | 62 ms | 1,433 MB |
mpnet-v2 is the highest-accuracy model in the family and the recommended default for production use.
Purpose
When a person searches on Connect211 they often express a need in plain, informal language — "I need a place to stay tonight", "help with my electric bill", "my kid needs shots". This model classifies those raw queries into one or more of 79 structured need categories (derived from the AIRS-aligned Gravity L2 taxonomy), enabling Connect211 to:
- Navigate its AIRS/211 service taxonomy automatically.
- Augment semantic search with category-level signals to surface more relevant results.
- Track population-level need trends over time.
Labels (79 categories)
The model was trained to predict all 79 unique categories in the Gravity L2 taxonomy. The gold standard evaluation below covers 55 of those 79 categories (spanning 18 real-world need scenarios). The remaining 24 categories were not included in the eval set but are fully supported by the model.
Evaluated (55):
Cash and Income Help Checkups and Screenings Child Care Programs
Childcare Cost Help Clothing and Apparel Criminal Justice Support
Daily Living Care Help Devices and Technology Digital Skills Training
Disability Support Help Disaster Response Help Drug Safety Programs
Education Programs Emergency Shelter Now Finding a Job
Food Benefits Programs Free Food Programs Free Health Supplies
Free Legal Services General Safety Resources Getting Free Rides
Getting Free Supplies Getting Healthcare Health Education Support
Help Paying Legal Costs Help Paying Utility Bills Help Paying for Care
Help Paying for Food Help Paying for Rides Help with Housing Costs
Help with Legal Costs Home Items and Goods Home Repairs and Safety
Housing Search Help Housing with Extra Support Internet Access Help
Job Skills Training Keeping Your Home Stable Learning New Skills
Legal System Navigation Low-Cost Housing Programs Lower Your Energy Costs
Medical Cost Help Mental Health Treatment Parenting and Family Help
Personal Finance Help Personal and Family Safety Phone and Internet Help
Stop Utility Shutoff Substance Use Resources Substance Use Treatment
Temporary Housing Help Transportation Cost Help Utility Bill Assistance
Vehicle Repair Help
Trained but not yet evaluated (24):
Consumer Rights Help Digital Access Assistance Disaster Preparedness
Disaster Recovery Help Education Financial Aid Funeral and Burial Help
Health and Safety Items Healthy Eating Support Help Paying for School
Keeping Your Home Safe Long-Term Care Services Mental Health Resources
Pet Care Assistance Physical and Speech Therapy Public Use Facilities
Safety Gear and Equipment School Success Support Small Business Support
Social Groups and Community Tax Preparation Help Travel Safety Programs
Unemployment Support Work Costs and Supplies Workplace Safety Help
Performance
All metrics were produced by a dedicated FastAPI inference server that wraps the fine-tuned SetFit model and applies a calibrated probability threshold per class. The server was containerised and load-tested independently; numbers below reflect end-to-end API latency, not raw model inference.
Accuracy — English gold standard (160 queries × 18 scenarios, 55 of 79 categories covered)
| Metric | mpnet-v2 |
|---|---|
| Gold hit rate (≥1 gold label in top predictions) | 74.4% |
| Top-1 exact match | 28.2% |
| Mean recall @ 3 | 46.1% |
| Mean recall @ 5 | 63.8% |
| Mean predicted labels per query | 6.3 |
| Mean over-prediction rate | 51.4% |
| Mean under-prediction rate | 30.2% |
| Mean probability cosine similarity (all 38 languages) | 0.647 |
Multilingual — translated gold standard (38 languages, 160 queries each)
| Metric | mpnet-v2 |
|---|---|
| Translated gold hit rate | 51.1% |
Languages evaluated include: Arabic, Amharic, Cantonese, Dari, Dutch, Fijian, Filipino, Finnish, French, German, Gujarati, Hawaiian, Hindi, Hmong, Haitian Creole, Japanese, Khmer, Korean, Lao, Malay, Mandarin (Simplified & Traditional), Nepali, Oromifaa, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Rwandan, Samoan, Somali, Spanish, Swahili, Tagalog, Ukrainian, Vietnamese.
Latency & resource usage (API server, containerised)
| Concurrency | p50 | p90 |
|---|---|---|
| 1 (single request) | 28 ms | 40 ms |
| 20 concurrent | 628 ms | 712 ms |
Idle memory: ~838 MB
Training
- Framework: SetFit (few-shot contrastive fine-tuning)
- Base model:
sentence-transformers/paraphrase-multilingual-mpnet-base-v2 - Task: Multi-label classification over 55 social-service need categories
- Training data: Synthetically generated labeled queries covering real-world search patterns observed on Connect211, augmented with human-reviewed examples
Usage
from setfit import SetFitModel
model = SetFitModel.from_pretrained("connect211/category-of-needs-classification-mpnet-v2")
predictions = model.predict(["I need help paying my rent", "looking for food bank near me"])
print(predictions)
About Connect211
Connect211 is a social-services search platform that helps individuals and caseworkers find health and human services programs across the United States. The 211 taxonomy is maintained in alignment with AIRS standards.